Think Together · Start with a Short Story
The girl cuts a sentence into little paper pieces. The robot says the small units it processes are called tokens, but they are not always whole words.
Lay Out the Process
- Original textA piece of writing
- TokenizerSplits it according to rules
- Token IDsUsed in model calculations
- After generationTurned back into text
A token is a unit that a language model uses to process text. It can be a word, part of a word, a character, or another piece.
The tokenizer decides how text is split, and different models may split it differently. For Chinese, do not assume that each character always equals one token.
The “paper pieces” comparison helps explain why the number of characters and the number of tokens are not always the same.
Now You Try
Does a long English word always use just one token?
Ready? Open this to see the explanation
Not necessarily. It may be split into several pieces, depending on the tokenizer's rules.
One Idea to Take Away
A token is a unit that a language model uses to process text. It can be a word, part of a word, a character, or another piece.
Try covering the diagram above and explaining each step to someone in your family. If one step is hard to explain, return to that card and look again.