Lesson 1.2: Setting up Python, NumPy and a tiny dataset
Set up a repeatable workspace for a language model: seeds, a fixed dataset, number formats, a stable softmax and fast array code, each verified by running it.
By the end of this lesson you will be able to build a clean, repeatable Python workspace for the course, fix a random seed so that every run gives the same numbers, verify that your dataset is byte for byte the one the lessons use, choose between 16, 32 and 64-bit numbers and say what each costs, write a softmax that does not overflow, and replace a slow Python loop with an array operation. You will show that the same seed returns the same random numbers and a different seed does not, that the corpus has the fingerprint 84eb9a58c009f4ff, that adding 0.1 a million times in 32-bit arithmetic drifts by 958 while the library's own sum drifts by 0.09, that a naive softmax on three scores near 1,000 returns "not a number" while a stable one returns 0.090, 0.245 and 0.665, and that a matrix product written as three loops runs thousands of times slower than the same product as one array operation. You should have read Lesson 1.1, and you should be comfortable with basic Python. This course is education about how language models work, not a guide to any commercial product.
A language model is a few hundred lines of code and a great many numbers. Most of the time you lose when building one is not spent on the clever parts. It goes on a result that changes between runs and cannot be reproduced, a loss that turns into nan halfway through training, a shape error that appears three layers after its cause, or a loop that takes an hour instead of a second. This lesson prepares for all four. None of it is deep, and all of it saves time.
The workspace
You need Python 3.9 or later and one package, NumPy. Nothing else is required for the first two modules, and later modules add at most a plotting library. Check your version and install NumPy into an isolated environment so that the course does not interfere with anything else on your machine:
python
# Run these in a terminal, not in Python:
# python -m venv llm-env
# source llm-env/bin/activate (on Windows: llm-env\Scripts\activate)
# pip install numpy
#
# Then, in Python:
import sys
import numpy as np
print(sys.version.split()[0], np.version)
On the machine used to check this course, that prints 3.11.5 1.26.4. Any recent versions behave the same for everything we do.
Keep one folder for the course with a simple layout: a file corpus.py containing the makecorpus function from Lesson 1.1, and a separate script for each lesson that does from corpus import makecorpus. Putting the data generator in one place means that every script sees the same data, and that if you change the generator, you change it everywhere.
Why NumPy and not PyTorch? NumPy performs arithmetic on arrays and nothing else. There is no hidden machinery for gradients, no device management, and no layers that do something you did not ask for. Everything that happens in a model we build is visible. If you later move to PyTorch, you will find that the array operations carry over almost unchanged.
Randomness you can repeat
A model begins with random numbers, and training samples random batches. If a result changes every time you run the script, you cannot tell whether a change in the code improved it. The fix is to use a seeded generator: a source of random numbers that starts from a known state, so that the sequence is the same every time.
python
a = np.random.defaultrng(0).standardnormal(3)
b = np.random.defaultrng(0).standardnormal(3)
c = np.random.defaultrng(1).standardnormal(3)
print(a) # [ 0.12573022 -0.13210486 0.64042265]
print(np.allclose(a, b)) # True: same seed, same numbers
print(np.allclose(a, c)) # False: different seed, different numbers
Two habits follow from this.
1. Create one generator and pass it around. Use rng = np.random.defaultrng(seed) and call rng.standardnormal(...), rng.choice(...) and so on, instead of the older global np.random.seed. A global seed can be disturbed by any library that draws a random number; a generator you own cannot.
2. Report the seed with the result. A loss of 1.04 with seed 0 and a loss of 1.07 with seed 1 are not different models. They are the same model, and the difference of 0.03 is noise from the random start. Before claiming that a change helps, check that it helps by more than that. We will measure this noise in Lesson 5.4.
A dataset you can trust
Every lesson uses the corpus from Lesson 1.1. To make sure that your copy is identical to the one used for the numbers in the text, take a fingerprint of it: a short code computed from every byte, which changes if even one character differs.
python
import hashlib
from corpus import makecorpus
text = makecorpus()
print(hashlib.sha256(text.encode()).hexdigest()[:16]) # 84eb9a58c009f4ff
print(len(text)) # 131158
words = text.split()
cut = int(0.9 len(words))
print(len(words), cut, len(words) - cut) # 29497 26547 2950
If your fingerprint is 84eb9a58c009f4ff and the three counts are 29,497, 26,547 and 2,950, then every number in the course will match yours. If not, the usual cause is a different NumPy random stream, a typo in a word list, or a different Python split. Compare the first sentences and fix it now. Debugging a model on subtly different data wastes days.
Two rules about the data will matter more as the course goes on:
Split in order, not at random. We take the first 90 per cent for training and the last 10 per cent for validation. Randomly shuffling sentences before splitting would put neighbouring text in both sets. For real documents, that lets the model see nearly the same passages in training and in the test, which makes the validation loss look better than it is.
Decide the split once, before looking at results. If you repeatedly adjust the model to improve the validation score, the validation set slowly becomes part of the training. Lesson 6.3 returns to this.
Numbers in a computer
A computer stores a real number in a fixed number of bits, so it stores an approximation. The format you choose decides how much memory you use, how fast the arithmetic is, and how accurate the answers are. These choices matter more in deep learning than almost anywhere else, because a model is millions of numbers added and multiplied billions of times.
Format · Bytes · Smallest relative step (epsilon) · Largest value · Typical usefloat64 · 8 · 2.2 × 10⁻¹⁶ · 1.8 × 10³⁰⁸ · Checking code, scientific work
float32 · 4 · 1.2 × 10⁻⁷ · 3.4 × 10³⁸ · Training, our default
float16 · 2 · 9.8 × 10⁻⁴ · 65,500 · Fast storage and inference
The epsilon is the smallest relative change the format can see: add something smaller than epsilon times a number to that number and nothing happens. Its consequences are visible without any machine learning. In float32, 0.1 is stored as 0.10000000149011612, not 0.1. Adding one to 16,777,216 changes nothing at all:
python
print(np.float32(16777216) + np.float32(1)) # 16777216.0
print(float(np.float32(0.1))) # 0.10000000149011612
The largest-value column is just as important. In float16, the exponential of 12 is already too large to store: it becomes inf (infinity). In float32, the exponential of 89 does the same. Because a language model constantly exponentiates scores to turn them into probabilities, large values and the limits of the format are the main source of failure.
Accumulating errors
Small errors in a sum build up. Add 0.1 one million times. The exact answer is 100,000.
python
s = np.float32(0)
for in range(1000000):
s += np.float32(0.1)
print(s) # 100958.34 (naive loop in float32)
print(np.sum(np.full(1000000, 0.1, np.float32))) # 100000.086 (NumPy's sum in float32)
print(np.sum(np.full(1000000, 0.1))) # 99999.9999999998 (float64)
The naive loop in float32 is wrong by 958, nearly 1 per cent. NumPy's own sum is wrong by 0.086, because it adds in pairs and so the running total never grows far larger than the numbers being added. In float64 the error is about 2 × 10⁻¹⁰. The same numbers, the same hardware, three different levels of accuracy.
The practical rules are these.
Use float32 as the default. It is accurate enough for training a small model and half the size of float64.
Sum with library functions, not loops. They are faster and more accurate.
Use float64 to check code. If a float32 and a float64 run give different results beyond the expected noise, there is a numerical problem to find.
Be careful with float16. Its range is so small that values overflow easily. In practice it is used for storing weights and for inference, and sometimes for training with extra safeguards. Module 8 returns to it.
A softmax that does not overflow
The most important function in a language model is the softmax. It converts V arbitrary scores, called logits, into V probabilities:
softmax(z)ᵢ = exp(zᵢ) / Σⱼ exp(zⱼ)
Written directly, it breaks. Take three scores near 1,000:
python
z = np.array([1000., 1001., 1002.])
naive = np.exp(z) / np.exp(z).sum()
print(naive) # [nan nan nan]
The exponential of 1,000 is far beyond the largest float64, so it becomes inf, and inf / inf is nan. The model's loss would become nan, every gradient after it would too, and training would be silently ruined.
The fix uses the fact that softmax does not change when the same number is subtracted from every score, because the factor exp(−c) appears in both the top and the bottom and cancels. Subtract the largest score, so that the largest exponent is exp(0) = 1 and nothing can overflow:
python
zz = z - z.max()
stable = np.exp(zz) / np.exp(zz).sum()
print(stable) # [0.09003057 0.24472847 0.66524096]
The three probabilities, 0.090, 0.245 and 0.665, are exactly the answer for the scores 0, 1 and 2, as the cancellation promised. The same trick gives the log-sum-exp, the logarithm of the denominator, which we need for the loss:
log Σ exp(zⱼ) = max(z) + log Σ exp(zⱼ − max(z))
For these scores that equals 1,002.4076. Lesson 1.4 uses it to compute the loss directly from logits without ever forming the probabilities, which is both safer and faster. Write the stable version once, test it with large inputs, and use it everywhere.
Arrays, shapes and broadcasting
NumPy is built around the array: a block of numbers with a shape, a tuple giving the size along each axis. A table of 6 numbers arranged in 2 rows and 3 columns has shape (2, 3). Every operation in a language model is a statement about shapes, and most bugs are shape bugs. Three habits catch nearly all of them.
Write the shape in a comment on every line that creates an array.
python
A = np.arange(6).reshape(2, 3) # (2, 3)
B = np.array([10, 20, 30]) # (3,)
print(A + B) # (2, 3) + (3,) -> (2, 3)
# [[10 21 32]
# [13 24 35]]
print(A @ B) # (2, 3) @ (3,) -> (2,)
# [ 80 260]
Know the broadcasting rule. When two arrays have different shapes, NumPy lines up their shapes from the right, and an axis of size 1 (or a missing axis) is stretched to match. That is why adding a vector of shape (3,) to a matrix of shape (2, 3) adds the vector to each row. It is also why (2, 3) + (2,) fails, as the sizes 3 and 2 do not match. Adding a bias to a layer's output, subtracting the largest logit in each row, and dividing each row by its sum are all broadcasting.
Assert shapes when it matters. A single line, assert logits.shape == (batch, vocab), turns a mystery three layers later into an error at the line that caused it.
The matrix product @ is the one operation we will use more than any other. Multiplying an (m, n) matrix by an (n, p) matrix gives an (m, p) matrix, where each entry is the dot product of a row of the first with a column of the second. It takes 2 × m × n × p arithmetic operations: a multiply and an add for each of the n terms in each of the m × p entries.
Why we vectorise
Python is slow at loops and NumPy is fast at arrays. A vectorised operation hands a whole array to compiled code in one call. The difference is not a few per cent.
Take two random 300 × 300 matrices and multiply them. Written with three nested loops, the product takes 54 million multiply-adds, one at a time, in Python. Written as W @ X, it is a single call:
python
import time
n = 300
rng = np.random.defaultrng(0)
W = rng.standardnormal((n, n))
X = rng.standardnormal((n, n))
t0 = time.perfcounter()
C = np.zeros((n, n))
for i in range(n):
for j in range(n):
s = 0.0
for k in range(n):
s += W[i, k] X[k, j]
C[i, j] = s
t1 = time.perfcounter()
D = W @ X
t2 = time.perfcounter()
print(np.allclose(C, D)) # True: the same answer
print(round(t1 - t0, 2), round(t2 - t1, 4)) # 7.72 0.0029 on the machine used here
print(round((t1 - t0) / (t2 - t1))) # 2683
On the machine used to check this course the loops took 7.7 seconds and the array operation took 0.0029 seconds, so the array version was about 2,700 times faster. Your timings will differ, and the ratio will too, but it will be in the thousands. Training a model needs trillions of operations. In loops, a one-minute training run would take days.
The rule is simple: if you write a Python for loop over the elements of an array, stop and look for the array operation. We will loop over training steps, over layers and over the words of a sentence, which are few. We will not loop over the entries of a matrix.
Use the calculator to see why arithmetic dominates the cost of larger models. The number of operations in one matrix product is 2 × m × n × p, and the time is that divided by the speed of the machine.
A health check for the course
Put these checks at the top of every script you write. They take a second to run and they catch the mistakes of this lesson before they cost you anything.
python
import numpy as np, hashlib
from corpus import makecorpus
def checkenvironment():
text = makecorpus()
assert hashlib.sha256(text.encode()).hexdigest()[:16] == "84eb9a58c009f4ff", "corpus differs"
words = text.split()
assert len(words) == 29497 and len(set(words)) == 34
z = np.array([1000.0, 1001.0, 1002.0])
p = np.exp(z - z.max()); p /= p.sum()
assert abs(p.sum() - 1) < 1e-12 and not np.isnan(p).any()
a = np.random.defaultrng(0).standardnormal(1)[0]
assert abs(a - 0.12573022) < 1e-8, "random generator differs"
return "environment ok"
print(checkenvironment())
If it prints environment ok, you have the same data, the same random stream and a softmax that cannot overflow.
Key takeaways
Use a virtual environment, NumPy, and one shared corpus.py. Everything in the early lessons uses nothing else.
Seed a generator you own with np.random.defaultrng(seed). Two runs with the same seed give identical numbers. Differences between seeds are noise, and an improvement needs to be bigger than that noise to be real.
Fingerprint the data. Ours is 84eb9a58c009f4ff: 131,158 characters, 29,497 tokens, split 26,547 for training and 2,950 for validation, in order.
float32 is the training default. A naive float32 loop adding 0.1 a million times drifts by 958, NumPy's sum by 0.09, and float64 by about 10⁻¹⁰.
Never compute a softmax without subtracting the largest score first. Scores near 1,000 give nan otherwise and 0.090, 0.245, 0.665 after the fix.
Write the shape in a comment, learn the broadcasting rule, and assert shapes when a mistake would be costly.
Replace element loops with array operations. The matrix product written as loops was about 2,700 times slower on our machine.
Common mistakes
Using the global random state. Another library can change it. Own a generator and pass it in.
Changing the data between experiments. A new dataset, a different split or a changed generator makes losses incomparable. Check the fingerprint.
Shuffling before splitting sequential data. Neighbouring text ends up in both sets and the validation score is too good.
Computing exp of raw scores. Subtract the maximum first, every time.
Trusting a loop sum in float32. Use the library function or float64.
Ignoring a shape that "works". Broadcasting can silently produce an answer of the wrong shape, such as a (5, 5) array where a (5,) array was meant. Print shapes while writing.
Check yourself
1. Why is np.random.defaultrng(0) better than a global seed?
The generator is an object you own and pass explicitly, so no other code can change its state. The global state is shared with every library that uses it.
2. float16 has a largest value of about 65,500. What happens to exp(12) in float16, and why does that matter for a language model?
exp(12) is about 162,755, more than 65,500, so it becomes inf. A model turns scores into probabilities with exp, so large scores would produce infinities, then nan.
3. What are the shapes of A + B and A @ B if A has shape (4, 5) and B has shape (5,)?
A + B is (4, 5), with B added to each row. A @ B is (4,), the dot product of each row of A with B.
4. The softmax of [5, 6, 7] and of [105, 106, 107] are equal. Why?
Adding 100 to every score multiplies every exponential by e¹⁰⁰. The factor cancels between the top and the bottom.
5. A matrix product of a (64, 512) matrix and a (512, 2048) matrix takes how many operations?
2 × 64 × 512 × 2048 = 134,217,728, about 134 million.
Try this
Set aside forty-five minutes. You need Python and NumPy.
1. Create the environment, save corpus.py, and run checkenvironment(). Fix anything that does not pass.
2. Run the sum experiment with 10 million terms instead of 1 million. How do the three errors change?
3. Compute the softmax of [0, 0, 0, 0] and of [0, 0, 0, 100] with the stable version. Describe the difference in plain words.
4. Time W @ X for n = 100, 200, 400 and 800. Plot or tabulate the times. By what factor does the time grow when n doubles, and does that match 2n³?
5. Write the stable log-sum-exp as a function and check it against np.log(np.exp(z).sum()) for small scores, and against the stable softmax for large ones.
6. Rewrite a loop that normalises each row of a 1,000 × 1,000 array to sum to 1 as one array expression using keepdims=True. Check that both agree.
A good result is a script that prints environment ok, a clear picture of where the numbers come from and where they can go wrong, and the habit of reaching for an array operation before a loop. Lesson 1.3 builds the central tool of the course: the machine that computes gradients.
Read more
David Goldberg's article "What Every Computer Scientist Should Know About Floating-Point Arithmetic" is the standard reference on the topic of this lesson. The NumPy documentation has a clear page on broadcasting, and the "NumPy: the absolute basics for beginners" guide covers the array operations used in this course. Andrej Karpathy's lecture on building a language model from scratch gives a good example of debugging shapes and numerical errors at the keyboard.
Tags: Build an LLM Python NumPy Floating point Reproducibility