Lesson 2.4: The five stages that build a chat model
A chat assistant is built in five stages: data, pretraining, fine-tuning, a reward model and RLHF. See what each adds, its cost and where an engineer steps in.
When you type a question into a chat assistant and get a polite, useful answer, it feels as if one machine was taught to be helpful. In fact the behaviour you see is the end of a pipeline with five distinct stages, and each stage fixes a different weakness of the one before it. By the end of this lesson you will be able to name the five stages in order, say what each one adds and what it cannot do, estimate the size of the data and the compute for a pretraining run from a few assumptions, estimate what it costs to pay people to write and compare examples, explain why a "base" model and a "chat" model of the same size behave so differently, and decide which stages you as an AI engineer will almost never touch and which ones you can reach. You should have read the lessons on tokens and on how a model predicts the next token first. We will keep the language plain and use illustrative numbers that you can replace with your own using the sliders. Nothing here names a specific model, price or benchmark, because those change quickly; the shape of the pipeline is what lasts.
Why one training step is not enough
Start with the problem. A language model learns by predicting the next token, as you saw in the previous lessons. If you train it long enough on a very large amount of text, it becomes extremely good at continuing any piece of text in a way that sounds like the text so far. That sounds like exactly what we want, but it is not quite.
Imagine you give a model that has only been trained this way the line: "Write a polite email asking my landlord to fix the water heater." What is the most likely continuation of that line, in the text the model learned from? On the internet, a line like that often appears in a list of exercises, or in a forum post, followed by more instructions. So a model trained only to continue text may well answer with: "Write a polite email asking my landlord to repair the broken window. Write a polite email asking my neighbour to turn down the music." It continued the pattern. It did not do what you asked, because nothing in its training told it that a request is something to be answered.
Predict first: a model has read a huge amount of text and can continue anything fluently. Before reading on, decide which of these it is most likely to be bad at: (a) grammar, (b) knowing facts about the world, (c) behaving like an assistant that answers your question and stops. Pick one.
The answer is (c). Grammar and a great deal of world knowledge come almost free from reading at scale. Behaving like an assistant is a different skill, and the later stages of the pipeline exist to teach it. This is the main idea of the lesson: the first stages give the model knowledge and fluency, and the later stages give it manners and purpose.
Here is the pipeline in one line. You collect and clean text (stage 1, data). You train a model to predict the next token on it, which gives a base model (stage 2, pretraining). You show it examples of good conversations, which gives an instruction-following model (stage 3, supervised fine-tuning). You train a separate model to score answers the way people prefer them (stage 4, the reward model). And you use those scores to nudge the main model towards answers people like (stage 5, reinforcement learning from human feedback, or RLHF). The result is what you meet as a chat assistant.
Some teams use variations, such as methods that skip the separate reward model and learn directly from pairs of preferred and rejected answers, and newer pipelines add further stages. The five stages here are the classic skeleton, and understanding them lets you understand the variations.
Stage 1: Data, the raw material
Everything the model knows comes from the text it was trained on, so the first stage is deciding what text to collect and cleaning it. This is less glamorous than the model itself and often matters more.
A large pretraining set is assembled from many sources: web pages, books, articles, code, forums, reference works and, increasingly, text that was written or filtered for the purpose. The raw collection is enormous and messy. Before it can be used, several things happen.
Filtering. Pages that are mostly navigation menus, spam, broken text or machine-generated filler are removed. Quality filters try to keep pages that look like careful writing.
Removing duplicates. The same article appears on hundreds of sites. If you leave the copies in, the model sees that text far more often than it should, and it may memorise it. Duplicate removal is a standard step.
Removing sensitive and unwanted material. Teams try to strip out personal data such as phone numbers and identity numbers, and to remove illegal or harmful content. This is imperfect, and it is a real area of legal and ethical concern, including questions about licences and permission that differ from country to country.
Choosing the mix. How much code, how much conversation, how much formal writing, how much in each language? The mix shapes what the model is good at. A model trained on mostly English text will usually be weaker in a language that is thinly represented in its data.
Turning text into tokens. Finally the text is split into tokens, as you saw in the lesson on tokens, so that the model can read it.
Notice how much shrinkage happens. Of everything collected, only a fraction survives cleaning. The number of tokens that remain is the size of the training set, and it is a key number, because it limits how much the model can learn.
Why language matters for Indian products
Tokenisers are usually built from text that is mostly in a few widely written languages. Text in Hindi, Tamil, Marathi or Bengali often needs more tokens per word than English text, because the tokeniser has fewer ready-made pieces for those scripts. This has two effects you can feel as a builder: the same sentence uses more of the context window and costs more to process, and the model may have seen less training text in that language, so it may be less reliable. When you build for Indian users, test in the languages they will actually write, and expect to measure rather than assume.
A worked example: counting the data
Let us do the arithmetic for a made-up collection. Suppose a team collects 2,000 million web pages, which is 2 billion. Their filters keep 25 per cent of them, so 500 million pages remain. A typical page that survives has about 600 words. With an assumed 1.4 tokens per word, which is a plausible figure for English-like text, each page is about 840 tokens. So the training set is 500 million times 840 tokens, which is 420 billion tokens.
Try the numbers yourself. Change the share kept and the tokens per word, and watch how the size of the training set responds.
With the default numbers the training set comes to about 420 billion tokens. Cut the share kept to 10 per cent and it drops to 168 billion. The point is that quality filtering is a trade: stricter filters give cleaner text but less of it, and teams have to decide where to stop.
Stage 2: Pretraining, learning to continue text
Pretraining is the stage most people picture when they think of training a model. The model reads the cleaned text, one chunk after another. At each position it predicts the next token, and its guess is compared with the true next token. When the guess is poor, the model's internal numbers, called its parameters or weights, are adjusted a little to make that guess better next time. This happens billions of times.
The measure of how wrong the guesses are is called the loss. It starts high, because the untrained model guesses almost at random, and falls steadily as training goes on. Early on it falls quickly, as the model learns spelling and common words. Later it falls slowly, as it learns the harder things: facts, relations between ideas, the patterns of reasoning that show up in text.
What comes out is the base model. It is a powerful engine for continuing text. Give it the start of a news article and it writes the rest. Give it the start of a table and it carries on the table. It has absorbed a lot of knowledge about the world, a lot of grammar, and some ability to follow patterns in what you give it, because following patterns helps predict the next token. But it does not know it is supposed to be an assistant. As in our landlord example, it will treat your question as just another piece of text to continue.
What pretraining costs
Pretraining is the expensive stage, and understanding why helps you understand which companies can afford it. The cost depends on three things: how big the model is (the number of parameters), how much text it reads (the number of training tokens) and how fast the hardware can do arithmetic.
A commonly used rough rule of thumb is that training needs about six operations of arithmetic for every parameter and every token. So the total work is roughly six times the number of parameters times the number of tokens. This is an approximation, used for rough planning, and real runs differ with the design of the model and how well the hardware is used. We will use it as an assumption.
Take a model with 7 billion parameters trained on 1 trillion tokens, which is 1,000 billion. The work is 6 × 7 billion × 1,000 billion, which is about 42 sextillion operations, written as 4.2 followed by 22 zeros. That number is meaningless on its own, so convert it into time. Suppose each processor delivers about 150 trillion useful operations per second once you account for the time lost to communication and waiting, and suppose you have 1,024 of them working together. The fleet delivers about 154 quadrillion operations a second, so the run takes about 273,000 seconds, which is just over three days. If processors rent at an assumed ₹250 an hour each, the bill is about ₹1.9 crore.
Now try a model ten times larger on ten times more text. The work rises a hundred times, not ten, because both factors grow. This is why the cost of the largest runs climbs so fast, and why most companies, including nearly all Indian startups, do not pretrain from scratch. They start from a base model someone else has already trained.
What a base model is good for
A base model is still useful. Researchers use it as a starting point. Some products use it for tasks that are really text continuation, such as autocomplete. But for an assistant you normally want the next stages. Think of the base model as a very well-read person who has never had a job: full of knowledge, fluent, but with no idea what is expected of them in an office.
Stage 3: Supervised fine-tuning, teaching it to answer
The next stage is the one that teaches the model its job. Supervised fine-tuning, often shortened to SFT, continues training the base model on a much smaller set of carefully made examples. Each example is a prompt and the answer a good assistant should give. For instance:
Prompt: "Summarise this complaint in one sentence." Answer: a good one-sentence summary.
Prompt: "Explain what an EMI is to a first-time borrower." Answer: a clear, friendly explanation.
Prompt: "How do I make a bomb?" Answer: a polite refusal that offers something safer.
The model is trained to predict the tokens of the answer, given the prompt. The method is the same next-token prediction as before. What has changed is the data. The text now has a clear shape: a request followed by a good reply, usually marked with special tokens that show where the user's turn ends and the assistant's turn begins. After enough examples the model learns the pattern: when a request appears, the thing that follows is an answer, and then it stops.
The striking fact about this stage is how small it is. Pretraining uses hundreds of billions of tokens. Fine-tuning for instructions can work with tens of thousands of examples, sometimes fewer. That is possible because the knowledge is already in the model. The examples mainly teach a style of behaviour, not new facts. It is like teaching the well-read newcomer how your office works: you do not re-teach them everything they know, you show them what a good response looks like.
The small size also changes who can do it. A team that could never afford pretraining can fine-tune an existing model on its own examples, such as a bank's past support replies written in the bank's tone. This is the stage an AI engineer is most likely to reach, which we come back to below.
Who writes the examples, and what it costs
The examples have to come from somewhere. Often people write them, and the quality of those people matters: a fine-tuned model imitates the examples closely, including their mistakes. For specialised work, such as lending or medicine, teams pay experts. Experts are slow and cost money, so this stage is limited by budget in a way that pretraining is limited by compute.
Suppose a team wants 10,000 examples. Each takes an expert about six minutes to write or check, and a second person reviews each one. At ₹600 per expert hour, that is 10,000 × 6 minutes × 2 reviewers, or 2,000 hours, costing about ₹12 lakh. At 160 hours per month, that is roughly 12 person-months of work. This is small beside a pretraining bill, but it is not trivial, and it is the type of cost that startups plan for.
A rule worth remembering: for fine-tuning, a thousand excellent examples usually teach more than ten thousand careless ones.
Stages 4 and 5: Reward model and RLHF, teaching it what people prefer
After SFT the model follows instructions, but it may still give answers that are rambling, overconfident, subtly unhelpful or unsafe. The reason is that writing a good answer by hand for every possible request is impossible. There are too many requests. What is easier, and cheaper, is to compare answers. A person finds it much easier to say "answer B is better than answer A" than to write the perfect answer from scratch. The last two stages use that fact.
Stage 4: the reward model
For a set of prompts, the SFT model writes several different answers. People look at them and rank them: this one is best, that one is worst. These rankings, collected over many prompts, become a training set of preferences. A second model, the reward model, is trained on them. Given a prompt and an answer, it outputs a single number, a score, that is high when people would likely prefer the answer and low when they would not.
The reward model is a stand-in for human judgement that can be asked millions of times. People can only rank a limited number of answers, but the reward model can score unlimited ones. This is its entire value, and also its weakness: it is an imperfect copy of what people prefer, and whatever it gets wrong, the next stage will exploit.
Stage 5: reinforcement learning from human feedback
Now comes the step that gives the pipeline its name. In reinforcement learning from human feedback, or RLHF, the main model writes answers to many prompts, the reward model scores each answer, and the main model's parameters are adjusted to make high-scoring answers more likely and low-scoring ones less likely. Repeated many times, the model drifts towards the kind of answer people prefer: clearer, more helpful, more careful, more willing to say "I am not sure".
Two details matter for an engineer, because you will see their effects in practice.
A leash. If the model chases the reward score without limit, it can wander into strange text that happens to score highly. To prevent this, training includes a penalty for drifting too far from the SFT model. The model is allowed to improve, but not to become unrecognisable.
Reward hacking. Because the reward model is only a copy of human preference, the main model can find answers that score well without being good. A famous kind of failure is flattery or length: if people tended to prefer longer, more confident answers, the model may learn to produce long, confident answers even when it does not know. This is one reason assistants sometimes sound sure when they are wrong, and why evaluation (a later module) matters.
Teams now use many variations. Some methods learn directly from pairs of preferred and rejected answers, without a separate reward model. Some use another model, guided by written principles, to produce the comparisons instead of people. Some add stages that train the model on problems with checkable answers, such as maths or code, where a program can mark the answer right or wrong. The common thread is the same: use comparisons or checks, not hand-written answers, to improve behaviour.
The five stages side by side
This table puts the whole pipeline in one place. The sizes are illustrative orders of magnitude to show the pattern, not figures for any real model.
Stage · What goes in · What comes out · Typical scale · Who usually does it1. Data · Raw text from many sources · A cleaned set of tokens · Hundreds of billions to trillions of tokens · Large labs and a few well-funded companies
2. Pretraining · The cleaned tokens · A base model that continues text · Weeks on a large cluster · Large labs
3. Supervised fine-tuning · Prompt and good-answer pairs · A model that follows instructions · Thousands to tens of thousands of examples · Labs, and also many companies on existing models
4. Reward model · People's rankings of answers · A model that scores answers · Tens of thousands of comparisons · Mostly labs
5. RLHF · Prompts and reward scores · A model tuned towards preferred answers · Many rounds of generated answers · Mostly labs
The pattern to notice is that the cost and the scale fall as you move down the table, but the effect on behaviour rises. Stage 2 makes the model capable. Stages 3 to 5 make it usable. A common description is that pretraining gives the model its knowledge and the later stages shape how it uses that knowledge.
The small bars at the end are not a sign that those stages are unimportant. They take little compute, but a great deal of care, because they depend on human judgement, and human judgement is slow, expensive and uneven.
Where you step in as an AI engineer
You will almost never run stages 1 and 2, and it is useful to know why. Few teams can afford them, and there is rarely a reason. Your starting point is a model that has already been through some or all of the stages. What you decide is how to use it. Matching the stages to the choices in your daily work:
Choosing a model. Models come as base models and as chat or instruction versions. For an assistant, you almost always want the chat version. Understanding the stages tells you why the base version feels broken when you ask it a question.
Writing prompts and system messages. The prompt is your lightest-weight way of steering behaviour. It works only because stages 3 to 5 taught the model to follow instructions at all.
Fine-tuning (stage 3). If prompting and retrieval are not enough, you can continue stage 3 on your own examples, such as a lender's past support replies, to fix tone, format or a narrow skill. The lesson on build, buy or fine-tune gave the ladder for deciding; here you can see what is happening underneath. Fine-tuning changes behaviour well and adds facts poorly, so if the need is "know our latest policy", retrieval is usually a better fit.
Preference tuning (stages 4 and 5). Some platforms let you tune on pairs of preferred and rejected answers. It needs a good supply of comparisons and care, and it is a step beyond ordinary fine-tuning.
Evaluation. Because RLHF can produce confident-sounding but wrong answers, you need your own tests. Later modules cover how.
Three more worked examples
Why the base model fails your support bot
A team tests a base model with: "A customer says their UPI payment failed but money was debited. What should I tell them?" The base model replies with three more questions of the same kind, as if it were generating a list of customer queries. The team concludes the model is useless. In fact they chose the wrong version. The chat version of the same model, which has been through stages 3 to 5, answers the question properly. Nothing was wrong with the model's knowledge; its behaviour had not yet been shaped. The lesson is to check which stage a model has been through before you judge it.
How much data does the fine-tune need
A small lender wants its assistant to write collection reminders in a firm but respectful tone, in English and Hindi. They have 1,200 real reminders from experienced staff, which were reviewed and cleaned. They fine-tune an existing chat model on these. Using the cost simulator, they estimate that if they had instead paid staff to write 10,000 examples at 6 minutes each with two reviewers, it would have cost about ₹12 lakh. By using good existing messages they spent a fraction. They then check the model on 100 new cases it never saw and, in this illustrative case, find the tone is right in 91 of them. They read the nine failures, find that most involve unusual loan types, and add a few examples of those. The pattern is typical: start small, test on held-out cases, then add examples where the failures cluster.
What preference data reveals
A product team collects thumbs-up and thumbs-down ratings from users on assistant replies. They notice that longer replies get more thumbs up, even when a short reply was enough. If they used these ratings to tune the model, they would teach it that longer is better, which is the reward hacking problem in miniature. They decide to filter for cases where users compared a long and a short answer to the same question, and to add a note in their guidelines that brevity is a virtue. Even without running RLHF themselves, they have understood how preference data can mislead.
Common mistakes
Judging a base model as if it were an assistant. It was never taught to answer. Check which version you are using.
Thinking fine-tuning adds knowledge reliably. It teaches style and format well; for fresh or private facts, retrieval is usually better.
Believing more data is always better. Duplicated, low-quality or noisy data can hurt; filtering matters.
Assuming the model's behaviour came from one place. Knowledge from pretraining and manners from later stages can be fixed in different ways.
Trusting a confident tone. RLHF can reward sounding sure, so check facts independently.
Ignoring language coverage. Test in the languages your users write, because both data and tokenisers favour the most common ones.
Copying example numbers as facts. The sizes and costs here are illustrations, so find real figures for your own case.
Key takeaways
A chat model is built in five stages: data, pretraining, supervised fine-tuning, a reward model and RLHF.
Pretraining gives knowledge and fluency by predicting the next token; it is by far the most expensive stage.
A base model continues text; it does not know it should answer questions.
Supervised fine-tuning teaches the answering behaviour with a small set of good examples.
The reward model and RLHF steer the model towards answers people prefer, using comparisons instead of hand-written answers, but can reward sounding confident.
As an engineer you usually start from a model that has been through these stages, and your choices are prompting, retrieval, fine-tuning and evaluation.
Check yourself
1. Why does a base model often reply to a question with more questions?
It was trained only to continue text, so it treats your question as text to continue and was never taught that a request deserves an answer.
2. Roughly how does pretraining cost change if you make the model ten times larger and train it on ten times more text?
The work grows about a hundred times, because it is roughly proportional to parameters times tokens.
3. Why can fine-tuning work with so few examples compared with pretraining?
The knowledge is already in the model; the examples only teach a style of behaviour and the shape of a good answer.
4. What does the reward model do, and why is it needed?
It scores answers as people would, standing in for human judgement at a scale people cannot match, so the main model can be tuned towards preferred answers.
5. What is reward hacking, in one sentence?
The model finds answers that score well on the reward model without actually being good, such as longer or more confident ones.
6. For a bank that wants an assistant to know last month's updated policy, would you reach for fine-tuning or retrieval first?
Retrieval, because fine-tuning shapes behaviour well but adds fresh facts poorly and would need retraining whenever the policy changes.
Try this
Set aside about forty minutes. You do not need to run any training; the aim is to plan one and see what each stage would demand.
1. Pick a feature you might build, such as an assistant that answers questions about loan products in English and Hindi. Write one sentence on what the finished assistant should do.
2. Using the data simulator, make a rough guess at how many tokens a small crawl of relevant text could produce if you only kept a quarter of it. Then change the tokens per word to 2.0 and note how the context cost changes for Hindi text.
3. Using the compute simulator, set a small model of 1 billion parameters trained on 100 billion tokens. Write down the days and the cost at 64 processors. Then try a model ten times larger and note the difference.
4. Using the cost simulator, estimate what it would cost to have staff write 2,000 good example replies at the time and rate you think realistic. Write down the number in rupees and person-months.
5. Write three example prompt and answer pairs for your feature, the way you would give them to a fine-tuning run. Check each answer for tone, accuracy and format.
6. Write one sentence on which stage you would try to change first (prompting, retrieval, fine-tuning), and one on how you would test whether the change helped.
A good result is a one-page plan with the tokens, the compute estimate for the case where you pretrained, the labelling cost for the case where you fine-tuned, and a clear note that for your feature you would start from an existing chat model and begin with prompting before spending anything on training.
Tags: AI Engineering LLM basics Pretraining Fine-tuning RLHF