An AI token is the smallest unit a language model splits text into before it starts calculating. Usually a token is a piece of a word: a short word in one go, a longer one in two or three parts, plus punctuation and spaces. Everything a model reads and writes is measured in this currency. Context windows, response times and invoices all hang off it.
For anyone working with AI this is not a footnote. Understanding how text becomes tokens explains why a document is “too long”, why the same text costs more in German than in English, and why a model fails at counting letters.
How text splits into tokens
Why not words and not letters
Both obvious approaches fail. A dictionary of whole words would be enormous and would still hit a gap at every typo, product name and piece of jargon. Individual letters, on the other hand, are so small that even short texts turn into endless sequences.
So models work with word fragments. A method such as byte pair encoding starts from single characters and merges the most frequent combinations step by step into larger building blocks, until a fixed vocabulary size is reached. Common words end up in the vocabulary whole, rare ones stay split. Nothing falls out: even a name never seen before can be assembled from characters.
German costs more
The rule of thumb for English is roughly one token per four characters, so about three quarters of a word. German comes off worse, for two reasons. Training data for most models is English-heavy, so German words appear in the vocabulary as whole units far less often. Then there are compounds. “Softwareentwicklungsprojekt” is one word, but certainly not one token.
In practice: the same content needs more tokens in German than in English. For languages with their own writing systems the surcharge is steeper still.
What else is expensive
Long runs of digits, base64 strings, minified code and emoji break into many small pieces because they barely occur in the vocabulary. A JSON file full of long IDs can burn more tokens than prose of the same length.
Where tokens show up in practice
| Term | Meaning | Practical consequence |
|---|---|---|
| Input tokens | Everything you send in: system instruction, history, attached documents | Grows quietly with every turn of a conversation |
| Output tokens | What the model produces | Billed at a higher rate than input by the common providers |
| Context window | Upper limit for input and output together | Once it is full, the oldest content drops out or the call fails |
| Tokens per second | Output speed | Determines how quickly an answer fills the screen |
Two misunderstandings are stubborn. First, the context window is not memory: it is refilled on every call, and whatever is not sent along is gone. Second, a large window is not a licence. A model answering over a brimful context is often less precise than one working from a short, well-chosen selection.
What drives consumption in a project
- The history. In a chat, the entire conversation so far is resent on every turn. The twentieth message costs a multiple of the first.
- The system instruction. It sits in front of every single call. Three paragraphs of rules, a thousand times a day, add up.
- Supplied documents. With retrieval augmented generation the retrieved passages land in the context. How many you allow is a trade-off between hit rate and cost.
- Tool descriptions. An AI agent is sent the definition of every available tool, on every round. With a connection over the Model Context Protocol that block grows with each additional server.
- Reasoning steps. Models that work through a problem at length before answering produce output tokens while doing so, even if you see little of it.
Saving without losing quality
The first step is always to measure. Nearly every API returns the token count with each call, and a look at the distribution usually reveals one outlier causing most of the bill.
After that a few unglamorous levers help: summarise the history instead of resending it in full, shorten and sharpen the system instruction (see prompt engineering), split documents into sensible sections beforehand rather than attaching whole PDFs, run recurring context blocks through the providers’ caching mechanisms, and use a smaller model for simple tasks. Where the output is highly structured, it pays to specify the format tightly. A model that only has to fill one field does not need to write an introduction.
Frequently asked questions about AI tokens
How many tokens does my text have? Exactly, only the tokeniser of the specific model can say, because each model family splits differently. For a rough estimate in English, four characters per token works; for German, count on roughly three, and expect to be on the low side.
Why can’t a model count letters? Because it does not see them individually. A word arrives as one or two blocks, not as a sequence of characters. Tasks like “how many Rs are in this word” ask for information that does not exist at the level the model operates on.
Are tokens and words the same thing? No, and the gap is wider than people expect. Short, frequent words are one token, long technical terms quickly four or five. Spaces and line breaks count too.
Does an image count as tokens as well? Yes. Multimodal models convert images into a token count that depends on resolution and level of detail. A high-resolution screenshot can occupy as much context as several pages of text.
Conclusion
Tokens are the unit of account behind every call to a large language model, and they explain a striking number of these systems’ quirks: the length limit, the cost curve, the failure at letter puzzles. Anyone building an AI feature into a product should measure consumption from day one rather than be surprised at the end of the month. How we wire such features into existing systems is described on our page about custom software development, and what actually holds up in projects in the article AI in software development. What a month of operation costs is something we work out before building, along with everything else that needs settling up front in an AI integration into existing software. For anything beyond that there is a free consultation.