Lesson 2.1: Tokenisation: characters, words and subwords
How text is cut into tokens, and why the cut matters: characters, words and subwords compared on vocabulary size, sequence length, unseen words and cost.
By the end of this lesson you will be able to explain what a tokenizer does, compare character, word and subword tokenisation on four measures (vocabulary size, tokens per word, handling of unseen words, and cost), measure all four on a text you generate yourself, and say why almost every modern language model uses subwords. You will show that a word tokenizer on our suffix-rich text needs 149 entries and fails on 5 of 33 words in new text (15 per cent), that a character tokenizer needs only 26 symbols but makes every sentence 4.4 times longer, which makes attention about 20 times more expensive, and that a subword tokenizer with 60 learned merges handles every word of the new text, at 1.87 tokens per word on the training text. You should have read Lessons 1.1 to 1.4. The code is plain Python. This course is education about how language models work, not a guide to any commercial product.
A language model does not read letters. It reads tokens: integers that stand for pieces of text. Everything the model can learn, say or misunderstand is shaped by how the text was cut into pieces before the model saw it. In Module 1 we cheated: our toy language had 34 words, and splitting on spaces gave a perfectly good set of tokens. Real text is not so kind. This module is about the step we skipped, and this lesson is about the decision that comes first: what should a token be?
What a tokenizer does
A tokenizer is a pair of functions.
encode(text) turns a string into a list of integer ids.
decode(ids) turns the list back into a string.
A good tokenizer is lossless: decode(encode(text)) returns the original text exactly. It also has a fixed vocabulary: the set of all pieces it knows, each with an id. The model's output layer has one score for every entry in the vocabulary, and its input table has one row for every entry, so the size of the vocabulary appears in the size of the model. Lesson 1.1 showed that for GPT-2 small, the token table was 31 per cent of all the parameters.
There are three obvious choices for the unit of text: a character, a word, or something in between.
A text with some structure
Our Module 1 text had no word forms: "chased" was one word and there was no "chasing". Real language builds words from parts, and tokenisation is mostly about that. So for this module we use a second made-up text that does. A generator writes sentences about twenty verbs, each in up to six forms, such as walk, walks, walked, walking, walker and walkers.
python
import numpy as np
def makecorpus2(nsentences=3000, seed=0):
rng = np.random.defaultrng(seed)
stems = ["walk","jump","play","help","paint","clean","cook","call","watch","talk",
"push","pull","kick","open","start","work","wait","look","fix","hunt"]
forms = ["", "s", "ed", "ing", "er", "ers"]
fw = np.array([0.15, 0.20, 0.25, 0.20, 0.12, 0.08])
sing = ["the boy", "the girl", "my aunt", "the old man", "our team", "the doctor"]
plur = ["the boys", "the girls", "my aunts", "the old men", "our teams", "the doctors"]
obj = ["the door", "the car", "a wall", "the box", "a green gate", "the small house"]
out = []
for in range(nsentences):
stem = rng.choice(stems); f = rng.choice(forms, p=fw); verb = stem + f
if f == "": s = f"{rng.choice(plur)} {verb} {rng.choice(obj)} ."
elif f == "s": s = f"{rng.choice(sing)} {verb} {rng.choice(obj)} ."
elif f == "ed": s = f"{rng.choice(sing + plur)} {verb} {rng.choice(obj)} ."
elif f == "ing": s = f"{rng.choice(sing)} was {verb} {rng.choice(obj)} ."
elif f == "er": s = f"the {verb} liked {rng.choice(obj)} ."
else: s = f"the {verb} saw {rng.choice(obj)} ."
out.append(s)
return "\n".join(out) + "\n"
The first lines it writes:
> the old man looks the car .
> the girls walk a green gate .
> the doctor was kicking the small house .
> the boys started the small house .
> our teams clean a wall .
The text has 3,000 sentences, 88,799 characters and 20,052 words. We keep the first 2,700 sentences for training: 18,042 words, 79,909 characters. The last 300 sentences are for validation. In training there are 149 different words and 26 different characters: 23 letters, the space, the full stop and the newline.
Choice 1: characters
The simplest tokenizer treats every character as a token. The vocabulary is the set of characters that appear: 26 here, a few hundred for a text in many languages, and a fixed 256 if we use the bytes of the text's UTF-8 encoding, which can represent anything.
python
chars = sorted(set(traintext))
stoi = {c: i for i, c in enumerate(chars)}
encode = lambda s: [stoi[c] for c in s]
decode = lambda ids: "".join(chars[i] for i in ids)
The strengths are clear. The vocabulary is tiny, so the token tables are tiny: with a width of 64 numbers per token, 26 symbols need 1,664 parameters. Nothing is ever unseen, since any string made from known characters can be encoded, and the model can learn spelling, because it sees every letter.
The weakness is length. The training text has 79,909 characters for 18,042 words, which is 4.43 tokens per word, counting the space after it. An average sentence is 29.6 tokens instead of 6.7. That matters more than it first appears, for two reasons.
1. The model must do more steps for the same content. To write one sentence, it takes 29.6 sequential predictions, not 6.7.
2. Attention gets expensive. In Module 3 we will see that each token looks at every earlier token in the context, so the work grows with the square of the length. A sentence of 29.6 tokens costs 29.6² = 876 units of attention, against 6.7² = 44.7 for a sentence of words. Characters make attention about 20 times more expensive for the same text.
A character model also has to learn that "w-a-l-k-i-n-g" is a word before it can learn anything about what the word means. Much of its capacity goes on spelling.
Choice 2: words
The opposite choice is one token per word. Split on spaces, give every distinct word an id, and each sentence is short: 6.7 tokens here. This is what we did in Module 1.
python
words = sorted(set(traintext.split()))
stoi = {w: i for i, w in enumerate(words)}
encode = lambda s: [stoi[w] for w in s.split()]
Word tokens are short and meaningful, and for a small, closed text they work. The vocabulary here is 149, with an embedding table of 149 × 64 = 9,536 parameters. But the approach breaks in three ways that matter for real text.
Unseen words. Suppose the model meets "mended", "dusted", "loading", "tracker" or "mends": five words whose stems were not in training. A word tokenizer has no id for them. We test this on a short new text of 33 words:
> the boy mended the door .
> the girls dusted a wall .
> our team was loading the car .
> the tracker saw the box .
> the old man mends the small house .
Five of the 33 words are unknown: 15.2 per cent. The usual repair is a special token <unk> that stands for "any word I don't know", but then the model cannot tell "mended" from "dusted", and every unknown word, however simple, is the same blank. Real text always contains new names, new spellings, typos, numbers and technical terms. A tokenizer that fails on them fails on text people write.
Forms are unrelated. To a word tokenizer, "walk", "walks", "walked", "walking", "walker" and "walkers" are six different ids with nothing in common. The model must learn each separately, and cannot carry what it learned about "walked" to "jumped". In English, much of that structure is regular, and in languages with richer word forms it is far more so.
The vocabulary grows without bound. Every new word needs a new row. English has hundreds of thousands of words, and across all languages, names and code, millions. A table of a million rows is too big, and most rows would be seen so rarely that the model could not learn them.
Choice 3: pieces in between
The solution used by essentially every modern model is to cut words into subwords: common words stay whole, rare words are split into pieces that occur often elsewhere. "Walked" might become "walk" + "ed", and a rare name might become several short pieces. In the extreme, a word is spelled out letter by letter, so any word can be written.
The pieces are not chosen by hand. They are learned from the text, by an algorithm that starts with single characters and repeatedly joins the pair that occurs most often. The next lesson writes that algorithm. Here we use its result to compare the three choices, with a small vocabulary of 60 learned merges, and then 150.
The learned subword tokenizer works on bytes, so it starts with 256 base entries and adds one per merge. After 60 merges the vocabulary has 316 entries, though only 66 of them occur in the training text, and the table of rows that matter is 66 × 64 = 4,224 parameters.
Property · Characters · Words · Subwords (60 merges) · Subwords (150 merges)Vocabulary entries used · 26 · 149 · 66 · 88
Tokens per sentence · 29.6 · 6.7 · 12.5 · 8.4
Tokens per word (training text) · 4.43 · 1.00 · 1.87 · 1.25
Tokens per word (new text, unseen verbs) · 4.4 · 1.0, with 5 unknown · 2.03 · 1.64
Unknown words in new text · 0 · 5 of 33 · 0 · 0
Attention cost per sentence (length squared) · 876 · 45 · 156 · 70
The subword tokenizers keep sentences short without the word tokenizer's failure. Consider how the unseen verbs are cut by the 150-merge tokenizer, with a leading space marked as part of the first piece:
New word · Pieces mended · men + d + ed
dusted · d + u + s + ted
loading · l + o + a + d + ing
tracker · t + r + a + c + k + er
mends · men + d + s
The tokenizer has never seen "loading" or "tracker", and has no entry for either. But it has learned ing and er as suffixes from the twenty known verbs, and spells the stems out letter by letter. The model receives something it can use: the stem as letters, and a meaningful ending. The unseen words have not been turned into a blank. This is the main reason subwords won.
The trade-off, as numbers
Subword vocabularies sit on a curve. With no merges, the tokenizer is a character (or byte) tokenizer, with the longest sequences and the smallest vocabulary. Each merge adds an entry to the table and shortens the text. In the limit, merging continues until every word is a token. The effect of merges on the training text:
Merges · Vocabulary · Tokens per word0 · 256 · 4.43
20 · 276 · 2.86
40 · 296 · 2.30
60 · 316 · 1.87
100 · 356 · 1.47
150 · 406 · 1.25
The gains shrink as the merges accumulate: the first 20 merges save 1.57 tokens per word and the last 50 save 0.22. The reason is that the algorithm joins the most frequent pair first, so later merges cover rarer sequences and shorten less text. Real tokenizers stop where the extra rows of the embedding table cost more than the shortening of sequences saves. GPT-2 stops at 50,000 merges (50,257 tokens including the 256 bytes and one special token), and recent models use between 100,000 and 250,000 entries, especially when they must cover many languages. For typical English text, such tokenizers use roughly 1.3 tokens per word. Text in languages and scripts that were rare in the tokenizer's training data needs many more, which makes it slower and more expensive to process and leaves the model less room in its context window.
Use the calculator to see how the three costs trade off for a text of a given length.
Bytes: a vocabulary with no gaps
One more design decision is how to build the starting alphabet. If it is the set of characters in the training text, an unusual character in new text, such as an accented letter or an emoji, has no id. Unicode has over 150,000 characters, too many for a base alphabet. The solution used by GPT-2 and most later models is to start from bytes. Every string can be written in UTF-8 as a sequence of bytes, each a number from 0 to 255, so a base alphabet of 256 entries can represent any text in any language, with no unknown character.
The price is that a non-English or rare character takes several bytes. For example, café ☕ is six characters but nine bytes: c, a, f, two bytes for é, a space, and three bytes for ☕. The ids come out as [99, 97, 102, 195, 169, 32, 226, 152, 149]. Learned merges make common characters in the training text into single tokens, and the rest remain as bytes. In the next lesson we write the algorithm and use the byte alphabet.
Special tokens
Tokenizers also reserve a few ids for control signals that are not text. The most common are an end-of-text token, which a model learns to emit when a document is finished and a generator can use as a stop signal, and, in chat models, markers for the start and end of each message and its speaker. They are added to the vocabulary by hand, appear in training data, and are never produced by merging. GPT-2's single special token is end-of-text, and it is the last of its 50,257 entries. Our toy models used the full stop as a crude signal of the end of a sentence; real systems use an explicit one.
What the tokenizer shapes
Because the model only ever sees tokens, the cut leaves visible marks on what it does well or badly.
Spelling and letter-level tasks. If "strawberry" is two or three tokens, the model never sees its letters directly. Counting the letter r in it needs knowledge that the model has to learn indirectly.
Arithmetic. If "1234" is one token and "1235" is another, unrelated one, the model cannot read digits as a place-value number without learning it. Many recent tokenizers split numbers into single digits or fixed groups for this reason.
Languages. Fewer tokens per word means more text fits in the context and costs less. A tokenizer trained mainly on English gives other languages more tokens per word.
Whitespace and code. Indentation, line ends and punctuation are tokens too. A tokenizer that handles runs of spaces well is better for code.
None of this is a flaw in the model's reasoning. It is a property of the representation that was handed to it, and knowing where it comes from helps in diagnosing odd behaviour.
Key takeaways
A tokenizer maps text to integer ids and back, and should be lossless. Its vocabulary size appears in the model's input and output tables.
Characters give a tiny vocabulary (26 symbols) and nothing unseen, but sentences are 4.43 tokens per word long and attention is about 20 times more expensive than for words (876 against 45).
Words give short sequences (6.7 tokens per sentence) and fail on unseen words: 5 of 33 words (15.2 per cent) in the new text, and every word form is an unrelated id.
Subwords learned by merging frequent pairs sit in between. With 60 merges the text takes 1.87 tokens per word, with 150 it takes 1.25, and nothing is unseen: "tracker" becomes t r a c k + er.
Each merge enlarges the vocabulary and shortens the text, with falling returns: the first 20 merges saved 1.57 tokens per word and the last 50 saved 0.22.
Starting from bytes gives a base alphabet of 256 that can represent any text: café ☕ is six characters and nine bytes.
The tokenizer shapes the model's strengths and weaknesses, in spelling, numbers, languages and code.
Common mistakes
Comparing losses across tokenisers. A loss per character, per word and per subword are in different units. To compare models with different tokenisers, convert to a common unit such as bits per character.
Building the vocabulary on all the text. Learn it on the training part only. Learning on validation text leaks information about it.
Using <unk> as a solution. It hides the problem. Prefer a tokenizer that can represent any input.
Ignoring the leading space. "the" and " the" are different strings. A tokenizer that attaches the space to the word keeps spacing exactly reversible.
Splitting on spaces for languages that don't use them. Some writing systems have no spaces between words. Byte-level subwords avoid the need to decide.
Changing the tokenizer after training. A model's tables are tied to its ids. A new tokenizer needs a new model.
Check yourself
1. A text has 1,000 words. A character tokenizer gives 5,400 tokens and a subword tokenizer gives 1,300. How much more does attention cost with characters?
Cost grows with the square of the length: (5,400 / 1,300)² = 4.15² = 17.3 times more.
2. A word tokenizer has a vocabulary of 50,000 and a width of 512. How many parameters does the input table hold?
50,000 × 512 = 25.6 million.
3. Why can a subword tokenizer represent a word it has never seen?
It can always fall back to smaller pieces, down to single bytes, so every string can be written as a sequence of known tokens.
4. héllo has five characters. How many bytes does it have in UTF-8, if é takes two?
Four single-byte letters and one two-byte letter: six bytes.
5. Why does adding more merges give smaller and smaller gains?
The algorithm joins the most frequent pair first. Each later merge covers a rarer sequence, so it shortens less of the text.
Try this
Set aside an hour. You need Python and NumPy.
1. Generate the text, split it into 2,700 and 300 sentences, and reproduce the character and word counts: 26 characters, 149 words, 4.43 characters per word, 6.68 words per sentence.
2. Build the word tokenizer with an <unk> entry. Encode the new text and decode it again. What do the five unknown words turn into?
3. Write down five more sentences with new verbs, such as "the cleaner mended a wall". What fraction of their words are unknown to the word tokenizer?
4. Compute len(text)2 and len(text.split())2 for a 1,000-character passage. How much cheaper is attention over words?
5. Take a sentence in a language you speak that is not English, and count its characters, words and UTF-8 bytes. What is the byte-to-word ratio?
6. Argue for a vocabulary size for a model that must handle English, code and numbers. What is your reasoning for the embedding table, and what for the sequence length?
A good result is the three tokenisers working on one text, a table of vocabulary against length against unseen words, and a feeling for why a middle path was chosen. The next lesson writes the algorithm that learns the pieces.
Read more
Rico Sennrich, Barry Haddow and Alexandra Birch's 2016 paper "Neural Machine Translation of Rare Words with Subword Units" introduced byte-pair encoding for neural models. Taku Kudo and John Richardson's 2018 paper describes SentencePiece, a widely used tokenizer library. The GPT-2 paper, "Language Models are Unsupervised Multitask Learners", explains the byte-level choice in section 2.2. Andrej Karpathy's lecture "Let's build the GPT Tokenizer" walks through the topic of the next lesson.
Tags: Build an LLM Tokenisation Subword tokens Vocabulary Byte-pair encoding