Build an LLM
8 stories tagged Build an LLM on Learning Fanatic.
- Lesson 1.2: Setting up Python, NumPy and a tiny dataset Set up a repeatable workspace for a language model: seeds, a fixed dataset, number formats, a stable softmax and fast array code, each verified by running it.
- Lesson 1.3: Tensors, gradients and autograd by hand Build a small automatic-differentiation engine in Python, check its gradients against numerical estimates, and use it to fit a line by gradient descent.
- Lesson 1.4: A first model: the bigram language model Train a bigram language model by gradient descent, check that it learns the probabilities a table of counts gives, and see what a one-word memory cannot do.
- Lesson 2.1: Tokenisation: characters, words and subwords How text is cut into tokens, and why the cut matters: characters, words and subwords compared on vocabulary size, sequence length, unseen words and cost.
- Lesson 2.2: Byte-pair encoding from scratch Write byte-pair encoding in about fifty lines: learn merges from text, encode and decode losslessly, and test it on unseen words and accented text.
- Lesson 2.3: Embeddings, vectors and token positions Give every token a learned vector, see similar words land close together, and add position information so a model knows word order, with code and numbers.
- Lesson 2.4: Context windows, batches and the data loader Turn a stream of token ids into inputs and targets, cut it into context windows, draw random batches, and see how batch size changes noise, coverage and memory.
- Lesson 1.1: What a language model is, and what we will build A language model gives the probability of the next token. See one as a table of counts, score it with loss and perplexity, and plan the GPT we will build.