This content has been automatically translated from Ukrainian.
When talking about ChatGPT and other large language models, the word “token” comes up constantly. The model has context for a certain number of tokens, the API counts input and output tokens, a long document can take tens of thousands of tokens, and an overly large conversation may eventually stop fitting entirely into context.
Because of this, it’s easy to imagine that a token is just a technical term for a word. But that’s not the case.
A token is a unit of information that the system breaks the text into before passing it to the language model.
A single short word can sometimes be one token, a longer one can be several, and punctuation marks, parts of words, numbers, and even fragments along with spaces can become separate tokens.
That’s why the number of words and the number of tokens in a text almost never match.
Why AI needs tokens at all
A person sees a sentence:
“Artificial intelligence helps to work with text.”
For us, it consists of words and has a clear meaning. The computer, on the other hand, must ultimately work with numbers.
Therefore, there is an additional stage between human text and the neural network — tokenization.
A special system called a tokenizer breaks the input text into certain fragments. Each such fragment corresponds to a numerical identifier. It is this sequence of numbers that is sent to the language model.
Conditionally, the process can be represented as follows:
text → tokens → numerical IDs → processing by the model → new tokens → response text
That is, when we write a question to ChatGPT, the model literally does not receive the sentence in the form we see it.
This is one of the basic details without which it is difficult to understand how modern language models work.
Is a token a word, a letter, or part of a word?
It can be all of the above.
Early natural language processing systems often worked with words. This approach seems logical: you can create a dictionary where each word corresponds to its own number.
The problem arises almost immediately. There are an incredibly large number of languages, and the number of possible words is practically unlimited.
In Ukrainian, for example, from the word “працювати” (to work) you can form “працюю” (I work), “працюєш” (you work), “працювали” (we worked), “працюватимемо” (we will work), and many other forms. This is further complicated by surnames, company names, slang, typos, new terms, URLs, programming code, and so on.
If every possible word is made a separate unit, a gigantic dictionary will be needed, which will never be complete.
The other extreme is breaking everything into individual characters. Then the dictionary will be small, but even a short sentence will turn into a very long sequence.
Modern language models often use a compromise — subword tokenization. Common words or fragments can remain whole, while rare ones are broken into smaller parts.
For example, the English ending -ing occurs so often that in BPE tokenizers it can exist as a useful recurring fragment. This is the principle that OpenAI cites in the documentation for its tokenizer tiktoken. At the same time, “subword” here does not mean that the tokenizer necessarily finds true roots, suffixes, or other morphemes. The division is formed statistically, and tiktoken uses a variant of BPE that works with bytes.
Something similar happens with Ukrainian words, although the specific division depends on the tokenizer. The same word in one model may consist of two tokens, while in another it may consist of one or three.
Therefore, an example like:
[нейро] [мережа]
can well explain the principle itself but does not mean that a specific model will necessarily break the word “нейромережа” (neural network) in exactly that way.
What is tokenization
Tokenization — is the process of converting raw data into a sequence of tokens that the model can work with.
For large language models, it is important to find a way of division that allows for a compact representation of common constructs while not getting lost when encountering unfamiliar words.
One of the most influential approaches has been Byte Pair Encoding, or BPE.
A significant role in the spread of subword tokenization in neural language systems was played by the work published in 2016 by Rico Sennrich, Barry Haddow, and Alexandra Birch “Neural Machine Translation of Rare Words with Subword Units”. The researchers showed how dividing into subword units helps neural machine translation work with rare and previously unknown words. Over time, this approach became extremely influential for NLP and systems like BERT and GPT.
In simplified form, the idea is this: the tokenizer learns which fragments frequently appear in the data and forms a dictionary of units convenient for compact text representation.
That’s why a popular short word may get one token, while a rare long word may break into several.
How a token turns into something understandable for the model
After tokenization, each token corresponds to a certain ID — an integer.
Conditionally:
“whale” → token № 18472
“sleeps” → token № 52931
These numbers are provided only to explain the principle — the actual IDs depend on the specific tokenizer.
The number itself does not contain the meaning of the word. Token № 18472 is not “more cat-like” than token № 18473 and does not mean that one token is more important than another. It is primarily an index by which the system finds the necessary representation.
Next, the token is transformed into a vector — a set of numbers that the model can mathematically process. This numerical representation is called embedding.
Therefore, a token and an embedding are not the same thing.
Token — the unit into which the text was broken.
Token ID — its number in the dictionary.
Embedding — the numerical representation that the neural network already works with.
Additionally, information about the token's position in the sequence and other model mechanisms is added. Then the Transformer analyzes the relationships between tokens and forms an internal representation of the context.
How ChatGPT generates text token by token
Tokens are needed not only for reading the query. The model's response is also created by them.
If we simplify greatly, the language model receives an existing sequence and determines the probabilities of possible next tokens.
For example:
“The capital of Ukraine is ...”
After that, the model evaluates possible continuations, and the generation mechanism determines the next token.
When it appears, the process repeats:
context → next token → updated context → next token → ...
Thus, a sentence, a paragraph, and a whole response are gradually formed.
Of course, a modern large language model is much more complex than a simple probability table. It takes into account a huge number of connections in the context, and modern systems can additionally use search, tools, file handling, and reasoning mechanisms. But at the level of text generation itself, the concept of a token remains fundamental.
How many characters or words does one token contain
There is no universal formula.
OpenAI provides a convenient approximate estimate for the English language: one token often corresponds to about four characters or three-quarters of a word. Accordingly, 100 tokens are approximately 75 English words. At the same time, the company separately emphasizes that the result depends on the language, context, model, and encoding.
Here, a particularly important clarification is — this formula applies to English.
It is not advisable to mechanically count Ukrainian texts according to it. Different scripts, morphology, and how effectively a specific tokenizer represents a given language affect the number of tokens.
Therefore, 1000 Ukrainian and 1000 English words do not necessarily take up the same number of tokens.
Moreover, even two almost identical lines can be tokenized differently due to case, space, or punctuation.
Why spaces and punctuation also matter
The tokenizer does not work with “concepts,” but with a specific sequence of characters or bytes.
Therefore:
“cat” “Cat” “ cat”and “cat.”
can form different sets of tokens.
In OpenAI's documentation, there is a telling example with the word red: its representation differs depending on whether it is capitalized, its position in the sentence, and whether there is a space before the word.
This may seem like a minor technical detail, but it explains a lot of unusual behavior of language models.
Why AI sometimes miscounts letters in words
A classic example is asking a language model to count the number of certain letters in a word.
For a human, this task is natural: we can look at the word character by character. For the language model, the initial unit is often not a single letter but a token that can contain several characters at once.
Therefore, a word that a person sees as a sequence of ten letters may be received by the model as several fragments.
This is one of the reasons why language models have historically found it more difficult to perform certain tasks with precise character counting, anagrams, or manipulations with individual letters than tasks that seem much more complex semantically.
Modern systems can compensate for this problem with reasoning, programming code, or other tools, so tokenization itself does not necessarily mean that the model will make a mistake. But it helps to understand where such a strange weakness comes from.
What is a context window and how do tokens relate to it
The term “context window” describes the amount of information that the model can consider when performing a task.
And this amount is measured in tokens.
The context can include not only the user's last question but also previous messages in the conversation, system instructions, parts of documents, results from tools, and other information necessary for the response.
Conditionally, if the model has a context window of 100,000 tokens, this does not mean that the user can insert text exactly 100,000 tokens long and receive an unlimited response on top. In many models, the total limit encompasses both input and output tokens, and the exact limitations depend on the specific model and service. Modern models with large contexts can support hundreds of thousands of tokens.
A large context window does not mean perfect memory
There is an important trap: if the model is technically capable of receiving a very long document, it does not mean that it will use each part equally well.
In a well-known study “Lost in the Middle: How Language Models Use Long Contexts”, the authors examined how language models find the necessary information in large contexts. In the models they studied, the result was often better when relevant information was located at the beginning or end of the context, and worsened when it was in the middle.
Since then, long-context models have significantly evolved, so the results of this work should not be automatically applied to every modern model. But the distinction between “can accept N tokens” and “equally well uses all N tokens” remains important.
Therefore, thoughtlessly adding dozens of documents to a query is not always better than providing a smaller but relevant set of information.
Context and memory are different things
Through tokens, it is also easier to understand the difference between context and AI memory.
Context is the information available to the model directly during a specific query. It can be imagined as a workspace: the model sees what is currently on it.
Memory in products like ChatGPT is a separate mechanism. Information from previous interactions can be stored outside the current context, and relevant details can be used in subsequent conversations as needed.
Thus, increasing the context window does not give AI infinite memory by itself.
What are input and output tokens
In developer services, you often see the concepts of input tokens and output tokens.
Input tokens — the information that the system has sent to the model for processing: the user's query and other context.
Output tokens — the tokens that the model generated in response.
In some APIs, there are other categories. For example, OpenAI separately accounts for cached tokens — input tokens obtained from the cache — and reasoning tokens, which some models use for internal steps before forming the final response. These categories are reflected in the API usage data.
For an ordinary person using ChatGPT through the website or app, these categories mostly remain a technical detail. For a developer working through the API, they directly affect resource usage and cost.
Why AI is often charged based on tokens
For APIs of large language models, tokens have become a convenient unit of measurement for usage.
A request for two sentences and an analysis of a 300-page document require completely different amounts of computation. The same applies to a short “yes” response and a detailed report of several thousand words.
Therefore, AI service providers often set prices based on the number of tokens processed.
In the OpenAI API, rates for language models, in particular, are set separately for input, cached input, and output tokens, and prices are usually indicated per million tokens. The cost depends on the specific model.
This should not be confused with a subscription to ChatGPT: a user on a regular ChatGPT plan does not receive a separate bill for each token used. The token-based pricing model is particularly important for developers integrating language models into their own applications via the API, as well as for specific corporate usage schemes.
Why a shorter prompt is not always better
If tokens affect context and cost, a logical question may arise: should all requests be made as short as possible?
Not necessarily.
Excessive twenty sentences may indeed be unnecessary. But an overly short request may lack the context needed for a good response.
For example:
“Write a letter.”
takes up very few tokens but explains almost nothing to the model.
In contrast:
“Write a short friendly letter to a client to inform them about rescheduling a meeting from Monday to Wednesday. Without an overly formal tone.”
will be longer but much more accurately defines the task.
Therefore, optimizing tokens is not an art of writing as little as possible. Rather, it is the skill of not wasting context on information that does not help accomplish the task.
Why different models count the same text differently
There is no single universal tokenizer for all artificial intelligence.
Different model families may have different dictionaries, tokenization algorithms, and text processing rules. Even within one company, different generations of models may use different encodings.
OpenAI explicitly states that the exact number of tokens depends on the model and encoding, and provides the tiktoken library for working with the tokenization of its models.
Therefore, the phrase “this text has 3000 tokens” without specifying the tokenizer is sometimes only approximate.
It is more accurate to say: “this text has approximately 3000 tokens for this particular model or encoding.”
What are special tokens
Not all tokens necessarily correspond to visible parts of regular text.
Models and systems around them may use special tokens that denote the structure of information: for example, the boundaries of certain fragments or service elements of the format. In tiktoken itself, there is also a separate mechanism for special tokens.
The user usually does not see them.
This is another reason why the number of tokens actually passed to the model cannot always be determined by simply counting the characters in the visible message.
Can tokens be non-textual
Yes. In multimodal systems, the concept of a token can apply not only to text.
If a model works with images, audio, or other types of data, this information also needs to be converted into a numerical representation suitable for processing by a neural network. The specific mechanism depends on the architecture: images can, for example, be processed through separate regions or patches and converted into a sequence of internal representations.
In modern OpenAI APIs, the use of multimodal models can indeed be accounted for by separate image tokens, audio tokens, and text tokens.
Therefore, in a broader technical context, the word “token” no longer always means just a piece of written word. But the way such units are formed for text, images, and audio can be completely different, so they should not be imagined as the same mechanism.
Tokens and model parameters are also not the same thing
Two more concepts that can easily be confused.
Tokens — are units of information that the model receives or generates.
Parameters — are numerical values inside the neural network itself, formed during the training process.
When it is said that a model was trained on trillions of tokens, it refers to the volume of training data in tokenized form.
When it is said that a model has billions of parameters, it refers to its internal structure.
That is, parameters belong to the model, while tokens pass through it.
Why the concept of a token is important for the ordinary user
For ordinary conversation with ChatGPT, knowing about tokenization is not necessary. One can use AI for years without ever counting tokens manually.
But this concept explains many things that otherwise seem strange.
Why can’t the model infinitely consider the entire conversation history? Because the context has a limited size in tokens.
Why is a huge document sometimes required to be split into parts? Because it must fit into the available context.
Why can one Ukrainian word take up more “space” than a similarly long English word? Because the model works not with words but with tokens of a specific tokenizer.
Why does using a language model via API have different costs for short and long tasks? Because the volume of processed and generated information is often measured in tokens.
And even the familiar phrase “AI reads text” looks a bit different after this. The language model does not see the page as we do. First, the sentence is broken down into small machine units, they are converted into numbers and numerical vectors, pass through the neural network — and only at the end does the sequence of generated tokens become words that appear on our screen.
