Lesson 1.1: What a language model is, and what we will build
A language model gives the probability of the next token. See one as a table of counts, score it with loss and perplexity, and plan the GPT we will build.
By the end of this lesson you will be able to define a language model as a function from a piece of text to a probability distribution over the next token, write a working one as a table of counts in about twenty lines of Python, measure it with cross-entropy loss and perplexity, and explain why a table stops working and a neural network is needed. You will show that a model that guesses at random on a 34-word vocabulary has a loss of 3.53 nats, that a one-word memory brings it to 1.16, that a two-word memory brings it to 0.99, which is within 0.004 of the best any model could do on our data, and that the same table idea would need 127 trillion cells for a real vocabulary and just three words of memory. You will also see where the 124 million parameters of GPT-2 small are, and what it costs to hold a model of that size in memory. No earlier lesson is needed. You should be able to read Python and know what a list, a dictionary and a logarithm are. This course is education about how language models work, not a guide to any commercial product.
This is the first lesson of a course with one purpose: to build a working GPT-style language model with our own hands, from a blank file, and to understand every number inside it. We will not call a library that hides the work. We will write the tokenizer, the attention layer, the training loop and the text generator ourselves, and at each step we will run the code and check that the numbers in the lesson are the numbers the code prints.
What a language model is
A language model is a machine that assigns a probability to text. More usefully for building one, it answers a single question: given the text so far, how likely is each possible next piece of text?
The "pieces" are called tokens. For now, treat a token as a word. Lesson 2.1 will show that real systems use fragments of words, but nothing in this lesson depends on the choice. The set of all tokens the model knows is its vocabulary, and its size is written V.
Given a context of tokens, the model returns V numbers that are non-negative and sum to 1: one probability per possible next token. After "the cat chased", a good model gives high probability to "the" and "a" and almost none to "." or "quickly". That is the whole interface. A model that answers this question well can do everything else we ask of it:
Generate text. Sample a token from the distribution, append it to the context, and ask again. Repeat. Every chat reply, every paragraph a model writes, is produced one token at a time in this loop.
Score text. The probability of a whole sentence is the product of the probabilities of its tokens, each given the ones before it. This is the chain rule of probability:
P(w₁, w₂, …, wₙ) = P(w₁) × P(w₂ | w₁) × P(w₃ | w₁, w₂) × … × P(wₙ | w₁, …, wₙ₋₁)
Answer, translate, summarise. If the context is a question, the likely continuation is an answer. The "ability" is the same prediction, applied to a context that contains a task.
The rest of this course, and nearly all of modern language modelling, is about making this one function better. The differences between a toy and a production system are the size of the context it can use, the number of parameters it can learn from, and the amount of data it has seen. They are differences of degree.
A world small enough to see
To learn anything about a model we need data where we know the right answer. Real text is too large and too messy for a first lesson, so we will use a small made-up language. A program writes sentences by picking a subject, a verb, an object, and, 60 per cent of the time, an adverb. Here is the whole generator. Every lesson in this course that uses it will paste this same function, so the data is always identical.
python
import numpy as np
def makecorpus(nsentences=4000, seed=0):
rng = np.random.defaultrng(seed)
subj = ["the cat", "the dog", "a bird", "the child", "my teacher", "the farmer", "a fox", "the baker"]
verb = ["chased", "watched", "found", "followed", "liked", "saw"]
obj = ["the ball", "a fish", "the red bus", "a small boat", "the old tree", "a loud drum"]
adv = ["quickly", "slowly", "today", "again", "at night"]
out = []
for in range(nsentences):
s = f"{rng.choice(subj)} {rng.choice(verb)} {rng.choice(obj)}"
if rng.random() < 0.6:
s += f" {rng.choice(adv)}"
out.append(s + " .")
return "\n".join(out) + "\n"
A few sentences from it, with the seed fixed at 0:
> a fox followed a small boat slowly .
> the cat chased a fish .
> a fox followed a small boat .
> the farmer followed a small boat .
> the baker liked the ball .
The full text has 131,158 characters and 29,497 word tokens, in a vocabulary of just 34 distinct tokens (the full stop counts as one). We keep the first 90 per cent, 26,547 tokens, for training, and the last 10 per cent, 2,950 tokens, for validation. The model learns from the training part only. We judge it on the validation part, which it has never seen, because a model that has memorised its training text has told us nothing about whether it has learned the language.
The reason for building the data ourselves is that we can work out the best possible score. The process is simple enough to calculate the uncertainty it contains: choosing a subject is 3 bits (one of eight), a verb and an object are each log₂ 6 = 2.585 bits, whether there is an adverb is 0.971 bits, and the adverb itself, in the 60 per cent of sentences that have one, is 0.6 × log₂ 5 = 1.393 bits. A sentence therefore carries 10.53 bits of surprise that no model can remove. With 7.387 tokens in an average sentence, that is 1.426 bits, or 0.988 nats, per token. A nat is a bit measured with the natural logarithm instead of base 2 (one nat is 1.4427 bits). We will use nats in this course because every library, and the calculus that trains the model, uses natural logarithms.
So 0.988 nats per token is the floor for this data. No model, however large, can get below it on average, and a model that reaches it has learned everything there is to learn. This makes the toy world a measuring stick. The models in the rest of this lesson, and in the whole course, are judged by how close they come.
The first model: a table of counts
The simplest language model is a lookup table. Look at the training text, count how often each token follows each context, and divide by the number of times the context appears. That gives a probability for each next token.
The context can be of any length. A unigram model uses none: it predicts each token by its overall frequency. A bigram model uses the one previous token. A trigram model uses the previous two. The general name is an n-gram model, where the context is the previous n − 1 tokens.
One detail matters: smoothing. If a context has never been seen in training, or a token has never followed it, the count is zero, the probability is zero, and a single zero makes the log-probability of the whole validation set infinite. The standard fix is to add a tiny count k to every cell before dividing, so that nothing is ever impossible. We use k = 0.01.
python
import math
from collections import Counter, defaultdict
text = makecorpus()
words = text.split()
vocab = sorted(set(words))
V = len(vocab)
cut = int(0.9 len(words))
train, val = words[:cut], words[cut:]
def ngramloss(order, k=0.01):
counts = defaultdict(Counter)
for i in range(order - 1, len(train)):
context = tuple(train[i - order + 1 : i])
counts[context][train[i]] += 1
total, n = 0.0, 0
for i in range(order - 1, len(val)):
context = tuple(val[i - order + 1 : i])
c = counts.get(context, Counter())
p = (c[val[i]] + k) / (sum(c.values()) + k V)
total += -math.log(p)
n += 1
return total / n, len(counts)
print("uniform guess", math.log(V))
for order in (1, 2, 3, 4, 5):
loss, ncontexts = ngramloss(order)
print(order, round(loss, 4), round(math.exp(loss), 3), ncontexts, V (order - 1))
The number this code prints, and that we will use for everything in this course, is the cross-entropy loss: the average of −ln P(correct token), over every position in the validation text. It is the surprise the model feels, on average, when it is shown what actually came next. A model that puts probability 1 on the right token has loss 0. A model that spreads probability evenly across V tokens has loss ln V.
Context length (tokens) · Model · Validation loss (nats per token) · Perplexity · Contexts seen · Possible contexts0 · Unigram · 3.184 · 24.14 · 1 · 1
1 · Bigram · 1.159 · 3.19 · 34 · 34
2 · Trigram · 0.992 · 2.70 · 123 · 1,156
3 · 4-gram · 1.000 · 2.72 · 292 · 39,304
4 · 5-gram · 1.035 · 2.82 · 769 · 1,336,336
A guess that treats every token as equally likely would score ln 34 = 3.526. The best possible score is 0.988. Read the table against those two numbers.
Three things stand out.
Context is worth a great deal. The unigram model barely improves on a blind guess, 3.18 against 3.53, because the frequency of a word tells you almost nothing about what comes next. A single previous token lets the model learn, for example, that "." is followed by a new sentence and that "chased" is followed by "the" or "a". The loss drops by two nats. A second token lets it learn that after "the red" only "bus" can come. In this world, the trigram is almost perfect.
More context eventually hurts. The 4-gram and 5-gram models do worse than the trigram, although in principle they contain more information. The last two columns show why. The 5-gram model has 769 distinct contexts that appeared in training, out of 1.3 million possible. Most of the contexts it meets in the validation text have been seen once or twice, or never, and the probabilities it computes from one or two observations are poor. A model with more freedom needs more data to pin that freedom down. This is the first appearance of a trade-off that will return in Module 5: capacity against data.
The score is a perplexity. The fourth column is e raised to the loss. It is the perplexity, and it has a simple reading: the model is as uncertain as if it were choosing uniformly among that many equally likely tokens. The unigram model, with a perplexity of 24.1, is as unsure as someone picking from about 24 words at each step. The trigram, at 2.70, is choosing among fewer than three. The best possible perplexity here is e^0.988 = 2.69. Perplexity is the loss in a more intuitive unit, and papers quote it for that reason.
Why a table cannot be the answer
The toy language has 34 words and the model needs 1,156 possible two-word contexts. A real vocabulary has about 50,000 tokens, which is the vocabulary of GPT-2. Count what a table would need:
A bigram table: 50,257 contexts, each with 50,257 next-token cells. That is 2.5 billion cells.
A trigram table: 50,257² contexts, each with 50,257 cells, which is 50,257³ = 126,937,424,324,593 cells, or 127 trillion.
Both are impossible to store. Worse, almost all the cells would be empty, because most three-word sequences are never seen, however much text you collect. And three words of context is nothing. To resolve what "it" refers to in a paragraph, or to carry the end of an argument back to its premise, the model needs hundreds or thousands of tokens of context. The number of possible contexts of that length is more than the number of atoms in the universe.
The table has two fatal flaws:
1. It cannot generalise. To a table, "the cat chased" and "the dog chased" are different contexts that share nothing. What it learned about cats it cannot apply to dogs. A model should notice that the two contexts are similar and give them similar predictions.
2. It grows without limit. Every extra token of context multiplies its size by V.
A neural language model replaces the table with a function that has a fixed number of adjustable numbers, called parameters, and that computes the distribution from the context instead of looking it up. Similar contexts give similar outputs, because they are processed by the same arithmetic. The number of parameters does not depend on how long the context is. The price is that the parameters must be found, not counted: we have to train the model, and training is the subject of Modules 4 and 5.
The model we will build
The model in this course is a small version of GPT, the architecture behind most current language models. "GPT" stands for generative pre-trained transformer. By Module 4 we will have written every part of it. Here is the plan in outline.
Module · What we build · The question it answers1 · Tools, autograd, a bigram model · How can a computer learn numbers by itself?
2 · Tokenizer, embeddings, data loader · How does text become numbers?
3 · Self-attention · How does a token use the ones before it?
4 · The transformer block and the full GPT · How are the parts put together?
5 · Loss, optimiser, training loop · How are the parameters found?
6 · Sampling and evaluation · How is text generated, and how good is it?
7 · Fine-tuning, LoRA, preference tuning · How does a base model become an assistant?
8 · Quantisation, serving, capstone · How is it made small and fast?
Everything is written in plain Python and NumPy. That is a deliberate choice. A framework such as PyTorch would shorten the code, but it would hide the things we want to see, the gradients and the matrix shapes. Each idea in the course will be written out once by hand so that you can see exactly what it does. Where a library would do the same job faster, we will say so. The code from Lesson 1.3 onwards includes a small automatic differentiation engine that we write ourselves, and you can check it against a numerical estimate of the gradient.
How big is "big"?
The count of parameters is the usual measure of a model's size. We can work it out exactly for GPT-2 small, which has 12 layers, a width of 768 numbers per token, a vocabulary of 50,257 and a context of 1,024 tokens. Do not worry about what every term means yet. By Lesson 4.4 you will derive each line.
Token embeddings: 50,257 × 768 = 38,597,376.
Position embeddings: 1,024 × 768 = 786,432.
One transformer layer: attention uses 4 × 768² + 4 × 768 = 2,362,368, the feed-forward network uses 2 × 768 × 3,072 + 3,072 + 768 = 4,722,432, and the two normalisation layers use 3,072. The total is 7,087,872.
Twelve layers: 12 × 7,087,872 = 85,054,464.
A final normalisation: 1,536.
The sum is 38,597,376 + 786,432 + 85,054,464 + 1,536 = 124,439,808, the "124 million" quoted for the model. Notice that 69 per cent of the parameters sit in the layers and 31 per cent in the table of token embeddings and positions. For a small model with a large vocabulary, the vocabulary is a big share of the cost.
Parameters of GPT-2 small by where they sit:
Parameters occupy memory, and training needs far more memory than the parameters alone. Each parameter is stored as a 32-bit number (4 bytes). Training also stores its gradient (4 bytes) and the two running averages that the Adam optimiser keeps (8 bytes), which is about 16 bytes per parameter before counting any of the intermediate results.
Model size · Weights in 32-bit · Weights in 16-bit · Training state at 16 bytes per parameter124 million (GPT-2 small) · 0.50 GB · 0.25 GB · 1.99 GB
7 billion · 28 GB · 14 GB · 112 GB
70 billion · 280 GB · 140 GB · 1,120 GB
A 124-million-parameter model trains on a laptop. A 70-billion-parameter model needs more than a terabyte for its training state alone, spread across many accelerators. Everything in the later lessons on serving and compression follows from this arithmetic.
The model that we build in the exercises will be much smaller than GPT-2: a few hundred thousand parameters, a vocabulary of a few dozen tokens, and training that takes a minute. It will be built from the same parts as GPT-2, and the same code with larger numbers would be GPT-2.
Use the calculator to size any GPT-style model. It uses the formula above: embeddings of V × d and T × d, 12d² + 13d for each layer, and 2d for the final normalisation.
From counts to a learned function
We now have the pieces of a plan. The table of counts is a language model in the full sense: it takes a context and returns a distribution. We have a way of scoring any model, which is the cross-entropy loss on held-out text. We know that the best possible loss on our data is 0.988 and that a table with a two-token memory comes within 0.004 of it. What remains is to replace the table with something that scales.
The next three lessons do that in steps. Lesson 1.2 sets up the tools. Lesson 1.3 builds the machinery that lets a computer find good parameters by itself, the gradient. Lesson 1.4 uses it to train a bigram model with parameters instead of a table, and shows that it learns the same probabilities as the counts. After that the model changes only in how it computes the distribution, and never in what it is asked to do.
Common mistakes
Judging a model on its training text. A table can reach a loss near zero on text it has memorised. Always score on held-out text.
Comparing losses across different vocabularies. A loss per character and a loss per word are not comparable. A model with a larger vocabulary pays more per token but has fewer tokens to predict.
Forgetting smoothing. One unseen pair gives a probability of zero and an infinite loss.
Treating perplexity as accuracy. A perplexity of 2.7 does not mean the model is right 37 per cent of the time. It is a measure of average uncertainty.
Assuming more context is always better. It is only better when the model, and the data, can make use of it.
Key takeaways
A language model maps a context to a probability for every possible next token. Generation, scoring and answering are all this one function used in different ways.
The loss is the average of −ln P(correct token) on held-out text, in nats. Perplexity is e to the loss, the number of equally likely options the model is choosing between.
In our toy language, a blind guess scores 3.53, a unigram model 3.18, a bigram 1.16 and a trigram 0.99. The best possible score is 0.988, which we can calculate because we wrote the generator.
Longer context helps only while there is data to fill it. The 4-gram and 5-gram tables scored 1.00 and 1.04, worse than the trigram.
A table of counts needs V to the power of the context length cells: 127 trillion for a three-token context with a 50,257-token vocabulary. It also cannot generalise from "cat" to "dog".
A neural network has a fixed number of parameters, computes the distribution instead of looking it up, and must be trained. GPT-2 small has 124,439,808 of them, needs 0.5 GB for the weights and about 2 GB to train.
In this course we write each part ourselves in plain Python and NumPy and check every number the lessons quote by running the code.
Check yourself
1. A model gives probability 0.25 to the correct token at every position. What is its loss and its perplexity?
Loss = −ln 0.25 = 1.386 nats. Perplexity = e^1.386 = 4. It is as unsure as a uniform choice among four tokens.
2. A vocabulary has 1,000 tokens. How many cells does a table for a four-token context need?
A four-token context means 1,000⁴ = 10¹² contexts, each with 1,000 next-token cells, so 10¹⁵ cells.
3. Why does the 5-gram table do worse than the trigram on the validation text, although it has more information?
Most of its contexts were seen once or never in training, so its probabilities rest on one or two observations and are noisy. A bigger table needs more data.
4. A bigram model scores 1.16 and the best possible score is 0.99. What does the gap of 0.17 nats mean?
On average the bigram assigns a probability about e^−0.17, or 16 per cent, lower to the correct token than the best model does. It cannot see the second previous token, which is the information it is missing.
5. A model has 2 billion parameters. How much memory do the weights alone need in 16-bit numbers?
2 × 10⁹ × 2 bytes = 4 GB.
Try this
Set aside an hour. You need Python 3 and NumPy.
1. Paste the corpus generator and the n-gram code into a file and run it. Check that your table matches the one above.
2. Change the smoothing k from 0.01 to 1.0 and to 0.0001. Which orders change most, and why?
3. Change the number of sentences in makecorpus from 4,000 to 400 and to 40,000. How does the trigram, 4-gram and 5-gram ranking change? What does this say about data and context?
4. Print the five most likely next tokens after the context ("the", "red") and after ("the",). Which model can tell them apart?
5. Write a function that generates a sentence from the trigram table by sampling one token at a time until it reaches a full stop. Print five sentences and judge them.
6. Work out by hand the best possible loss for a language where every sentence is "a b c ." with a, b and c each chosen uniformly from four options. Then check it by counting.
A good result is a table of losses that matches ours, a feel for how the context length and the amount of data interact, and a first generator that produces plausible sentences. The next lesson sets up the tools we will use for the rest of the course.
Read more
Claude Shannon's 1948 paper "A Mathematical Theory of Communication" introduced the entropy of text and the n-gram approximations to English that this lesson uses. Daniel Jurafsky and James Martin's textbook Speech and Language Processing has a chapter on n-gram models and smoothing. Andrej Karpathy's video series "Neural Networks: Zero to Hero" builds a character-level model in the same spirit as this course. The GPT-2 parameter count can be checked in the model card that OpenAI published with the 2019 release.
Tags: Build an LLM Language models Next-token prediction N-gram models Perplexity