Token
In brief
A token is the unit of text (word, word fragment, punctuation) that a language model handles; it is also the unit used to measure the length and cost of a request.
Sources[1]
Definition
A token is a unit of text that the model reads and produces. Depending on the tokeniser, it can be a whole word, a word fragment, a single character or a punctuation mark. Text is split into tokens, and each token is converted into a numeric identifier.
Why not whole words?
Modern tokenisers use subwords (for example via the Byte Pair Encoding algorithm, adapted to language processing in the paper by Sennrich et al.). This makes it possible to represent rare or unknown words by breaking them into known fragments, with a vocabulary of reasonable size.
Why it matters
- Length: a model’s context window is measured in tokens, not words.
- Cost: API access to models is generally billed by the number of tokens processed (input and output); consumer subscriptions, by contrast, are usually flat-rate, with usage limits.
- Languages: the same text may need more tokens depending on the language and the tokeniser, which affects the cost and the space available in the context.
Rough figures
There is no universal conversion between words and tokens: it depends on the tokeniser and the language. For a reliable figure, use the counting tool provided by the model’s vendor.
Token calculator
Type or paste some text: see how many tokens it represents and how the model splits it.
- Tokens
- …
- Characters
- 98
- Words
- 16
- Characters per token
- –
That is 0% of a 128,000-token context window.
Token breakdown
Each colour is one token. Spaces are often part of the token that follows.
Loading the tokenizer…
Exact count for the o200k tokenizer used by GPT-4o and later OpenAI models. Claude, Gemini, Mistral or Llama use their own tokenizers, so their totals may differ noticeably. Everything runs in your browser; your text is never sent anywhere.
Spotted a mistake or have an idea about this entry? Let us know