Lesson 2.2: How a model predicts the next token
A language model works by predicting one token at a time. Understand probabilities, why answers vary, and why fluent text is not the same as true text.
Underneath every chatbot answer is a simple repeated step: given the text so far, predict what token comes next. Understanding this one idea explains why models can write fluent paragraphs, why they sometimes state things that are false, and why asking the same question twice can give different answers. By the end of this lesson you will be able to describe in plain words how a model produces an answer, read a table of next-token probabilities, explain why fluent text is not the same as true text, and use two calculators to see how often repeated runs agree and how quickly errors accumulate in an answer that contains several facts. You need no mathematics beyond percentages. If you have read the previous lesson on tokens, you already know the unit the model works in.
This lesson is the foundation for the next one, on temperature, which is the setting that controls how the model picks from its probabilities. It also explains, from the inside, many of the design choices you met earlier: why we give the model facts in the request, why we check outputs and why we measure with many examples and not one.
One step, repeated
Imagine the sentence "The capital of India is". A model looks at these tokens and produces a probability for every token in its vocabulary, a list of tens of thousands of possible next pieces of text, each with a number saying how likely it is to come next. "New" might get a very high probability, and a stray word might get almost none. The model picks one token, adds it to the text, and then repeats the whole process for the next token, now looking at the longer text.
That is the whole mechanism. There is no separate step where the model "decides what it wants to say" and then writes it. The answer emerges one token at a time, and each token is chosen on the basis of everything before it, including the tokens the model itself just produced.
An answer of 100 words is therefore built by about 130 small predictions, using the rough rule that 100 tokens is 75 words. Each prediction is a fresh choice from a fresh list of probabilities. Two consequences matter for engineers. First, the model does not plan the whole answer ahead the way a person writing an essay might, although it can produce well-structured text because the patterns it learned include structure. Second, a slip early in an answer can send everything after it in a different direction, because later tokens are chosen to fit what is already written, even if what is already written was a mistake.
Where the probabilities come from
The probabilities are not looked up in a table that someone typed in. They come from training. The model was shown an enormous amount of text and repeatedly asked to predict the next token, and every time it was wrong its internal numbers were adjusted to make the right continuation a little more likely. After a very large number of such adjustments, the model became good at the pattern "what usually follows text like this".
This explains both the strength and the weakness of the method. The strength: to predict text well, the model had to pick up a great deal about grammar, facts, styles of writing, how arguments go and how code is laid out, so it can do many tasks it was never explicitly taught. The weakness: what it learned is what text usually looks like, which is related to truth but is not the same thing. A statement that is common in the training text is easy to produce, whether or not it is correct. A statement that is true but rare, or about something that happened after training, is harder.
Here is a toy example of what the model's raw output looks like at one step. Suppose the text so far is "The meeting is on", and the model gives these probabilities for the next token. The numbers are illustrative.
The model has not decided that the meeting is on Monday. It has produced a spread of possibilities, with Monday most likely. What happens next depends on how the application chooses from the spread, which is the subject of the next lesson. For now, notice that nothing in this list is "the answer". It is a distribution, and the model's output is a draw from it.
Why fluent is not the same as true
The model has learned patterns from huge amounts of text. It is very good at producing text that sounds like what usually follows. It does not look facts up in a table. When the training patterns strongly point to one answer, the result is usually correct: ask for the capital of India and every pattern agrees. When they do not, the model still produces a confident-sounding continuation, because producing a continuation is the only thing it does. It has no built-in separate signal that says "I am guessing here". This is the source of what people call hallucination: a plausible sentence that happens to be wrong.
Hallucinations come in a few recognisable kinds.
Invented specifics. A reference number, a court case, a quotation or a web address that looks right and does not exist. The shape of such things is easy to imitate and the detail is not.
Out-of-date facts. The model learned up to some date and knows nothing later. It will describe a fee or a rule as of its training, in the present tense.
Fluent mistakes in reasoning or arithmetic. The steps read well and the number is wrong, because the model produces what a good calculation usually looks like and does not carry it out.
Facts about your own business. It has never seen your policy, so it fills the gap with what policies like yours usually say.
Engineers handle this by giving the model the facts it needs in the request, and by checking answers, rather than trusting its memory.
Predict first: you ask a model for the reference number of a specific government notice it has never seen. Will it more likely say "I do not know" or produce a plausible-looking number? Decide before reading on.
Often it produces a plausible number, because plausible-looking text is what its training rewards. Some models are trained to admit uncertainty, but you should never rely on that. Provide the notice, or verify the number yourself.
Why answers can differ
Because the model works with probabilities, it does not have to pick the single most likely token every time. Most applications let it choose among several likely tokens, which makes writing feel natural and varied. It also means two runs of the same question can differ. The next lesson shows how to control this.
Use the sampler below to compare always taking the top token with sampling, on a case where the model is unsure and one where it is sure.
How much variation should you expect? It depends on how confident the model is. When one continuation has a very high probability, almost every run takes it and the answers are nearly identical. When the probabilities are spread out, different runs go different ways, and because each token feeds into the next, small differences early can grow into quite different answers later.
The simulator below makes this concrete. Choose the probability of the most likely answer and the number of times you ask the same question, and it shows how likely it is that every run gives the top answer. The chart shows how that chance shrinks as you ask more times.
At the starting numbers, three runs all give the top answer only about 12 times in 100, so almost nine times in ten you will see at least one different answer. Now raise the probability to 95 per cent and ask five times: the chance that all five agree is about 77 per cent. This is a good intuition to carry. A system's output is only stable where the model is confident, and one test run is not evidence of anything. It is why later lessons stress building a test set and running it many times.
A worked example: the meeting day
Suppose after "The meeting is on" the model gives these probabilities: "Monday" 50 per cent, "Tuesday" 30 per cent, "Friday" 15 per cent, anything else 5 per cent. If it always picked the top choice, it would say Monday every time. If it samples, about half the runs say Monday, a third say Tuesday, and so on. Neither is more right: the model has no calendar, so only the context you provide can fix the answer.
Put the real meeting day into the request, "The meeting is on Thursday. When is the meeting?", and the probabilities shift sharply: "Thursday" now gets nearly all the probability, because the text before it makes that continuation overwhelmingly likely. This is the whole logic of context building: you do not make the model know things, you make the right continuation the likely one.
Errors add up inside an answer
There is a second piece of arithmetic that matters, and it concerns how a single answer can contain many claims. A typical answer to a business question holds several checkable facts: a date, an amount, a name, a rule. Suppose each is right 95 times in 100. That sounds reliable. But for the answer to be wholly right, every one of them must be right, and the chance of that is the product of the individual chances. With eight facts, it is about two answers in three. The other third contain at least one wrong fact.
This is why checking matters so much in practice. A check, whether code that compares a number with a database or a person who reads the output, can catch some of the wrong facts before they reach the user. Even a check that catches most errors changes the outcome greatly.
Try the simulator below. Set how many checkable facts the answer contains, how often each is right and how many of the wrong ones your checks catch. The chart compares answers that reach the user with at least one wrong fact, with and without your checks.
Run three experiments. First, with no checks, move the number of facts from 8 up to 20 and watch the share of flawed answers climb to about two in three. Second, set the checks to catch 80 per cent of errors at 8 facts: the share that ships a wrong fact drops from about 34 per cent to under 8. Third, instead of checking, raise the chance each fact is right from 95 to 99 per cent and see what that buys. Often the checks are the cheaper lever.
Odd failures that follow from the design
Once you see that the model works with tokens and with likely continuations, several behaviours that look strange stop being mysterious.
Spelling and counting letters. Ask a model how many times a letter appears in a long word and it may get it wrong. The reason is that it does not see letters. It sees tokens, which are chunks, so the individual letters inside a chunk are not directly visible to it. It has to work out spelling from patterns it learned, and that is unreliable. For a task like this, ordinary code is the right tool: count the letters with a program and let the model handle language.
Arithmetic. Multiplying two long numbers is a task where the likely-looking answer and the correct answer can differ. The model produces digits that resemble a correct result, and for small sums it is often right, because those patterns were everywhere in its training text. For long or unusual calculations it can slip. Again the engineering answer is to let code do the arithmetic and have the model call it, a pattern you will meet in the module on tools.
Being talked into an answer. If your question contains a false assumption, such as "why did the company cut its dividend last year?" when it did not, the most likely continuation is an answer that goes along with the assumption. The model is completing text that sounds like a reasonable reply to the question. Neutral wording, and checking against real data, reduces this.
Stating things differently when asked differently. A small change in phrasing changes the text the model is continuing, so it can change the answer. This is not carelessness. It is the same mechanism working on different input, and it is why you test with varied wording.
Step-by-step reasoning. Asking a model to work through a problem in steps often improves results on tasks that need several stages. One way to understand this is that every token it writes becomes part of the text it conditions on, so writing intermediate steps gives it more relevant text to continue from. It is not a guarantee, and a fluent chain of steps can still end in a wrong answer, so the result needs checking like any other.
The common thread is that the model is a very capable continuer of text and not a calculator, a database or a witness. Use it for what it does well, language, summary, drafting, classification and explanation, and give the exact parts of the job to systems that are exact.
What engineers do about it
The next-token idea points directly to the practices you will meet in the rest of this track.
Put the facts in the request. If the right information is in front of the model, the correct continuation becomes the likely one. This is retrieval, covered in the next modules.
Ask the model to stay within the text. An instruction such as "answer only from the document, and say so if the answer is not there" shifts the probabilities toward admitting a gap, although it does not remove the risk.
Check what can be checked. Compare amounts with databases, dates with calendars, quotations with source text and totals with their parts.
Keep answers focused. Fewer claims per answer means fewer chances for one to be wrong.
Control the variation. For tasks that need the same answer every time, such as extraction, reduce the randomness, which is the subject of the next lesson.
Test on many examples. One run proves nothing. A set of examples, run repeatedly, shows how the system really behaves.
Keep a person in the loop where mistakes are costly. Money, health, legal rights and safety are not the places to rely on a fluent guess.
Common mistakes
These are the misunderstandings that most often lead people astray when they start using models.
Treating fluent text as true text. Confidence of tone says nothing about accuracy.
Trusting the model's memory for facts about your business. It has not seen them. Supply them.
Judging a system from one run. The same request can give different answers, so test many.
Assuming the model plans the whole answer. It writes one token at a time, and an early slip can carry forward.
Asking for many facts at once and expecting all to be right. Reliability falls as the number of claims rises.
Believing a citation because it looks right. Invented references look exactly like real ones. Check that the source exists and says what is claimed.
Relying on the model to say "I do not know". Some models do, but not reliably. Design for the case where it does not.
Key takeaways
A model generates text by predicting one token at a time, each based on everything before it.
It produces probabilities, then chooses a token, so fluent text is not guaranteed to be true.
Hallucination happens when patterns do not support a clear answer but the model still continues confidently.
The same question can give different answers, and variation is greatest where the model is least sure.
An answer with many facts is only as reliable as all of them together, so reliability falls as claims increase.
Give the model the facts it needs, check what can be checked and test across many examples.
Check yourself
1. In one sentence, how does a model produce a 100-word answer?
By predicting one token at a time, about 130 times, each prediction based on all the text before it.
2. Why might a model give a confident but wrong reference number?
Plausible-looking text is what its training rewards, and it has no separate signal for when it is guessing.
3. The top answer has a 60 per cent chance. About how likely is it that two runs both give it?
About 36 per cent, because 0.6 times 0.6 is 0.36.
4. An answer holds 5 checkable facts, each right 90 per cent of the time. About how often is the whole answer right?
About 59 per cent, because 0.9 to the power of five is about 0.59.
5. Why does putting the real meeting day in the request fix the answer?
It makes the correct continuation overwhelmingly likely, so the model's probabilities concentrate on it.
Try this
Set aside about forty minutes and use any chat model you have access to.
1. Pick a factual question where you know the answer, and ask it three times. Note whether the three answers differ in wording or in substance.
2. Ask a question about something very specific it is unlikely to know, such as an internal document of your own. Note what it does when it cannot know.
3. Paste in the correct source text and ask again. Write down what changed.
4. Count the checkable facts in one of its longer answers and mark each right or wrong.
5. Put your counts into the checking simulator above and see how likely an answer of that kind would be to be fully right, and how much a check would help.
6. Write down one check, by code or by a person, that would catch the kind of error you saw.
A good result is a short note with three observations: how stable the answers were, what happened without the facts and what changed with them, plus one check you would add. It is a small evaluation, which is exactly what a real project would begin with.
Tags: AI Engineering LLM basics Next token prediction Probability Beginner