Lesson 2.3: Temperature and sampling explained
Temperature decides how adventurous a model's word choices are. Move the slider to see the probabilities reshape, and learn which setting fits which task.
In the last lesson you saw that a model gives every possible next word a probability, then picks one. Temperature is the dial that controls how that pick is made. It is one of the few settings you will change on almost every project, so it is worth seeing exactly what it does to the numbers. By the end of this lesson you will be able to explain what temperature does in plain words, predict what happens to the choice of words when you raise or lower it, choose a sensible starting setting for tasks such as extraction, summaries and brainstorming, understand why even a low setting does not guarantee identical answers, and test a setting properly instead of guessing. You need only the idea from the previous lesson that a model scores many possible next words. All the numbers here are small made-up examples, so you can follow the arithmetic.
Scores first, probabilities second
Inside the model, each candidate word gets a raw score. A higher score means a better fit. Those scores are turned into probabilities that add up to 100 percent. Temperature enters at that last step: the scores are divided by the temperature before the conversion.
Low temperature (below 1) exaggerates the gaps. The best word pulls far ahead and the others shrink.
Temperature of 1 leaves the model's own probabilities as they are.
High temperature (above 1) flattens the gaps. Weaker words get a bigger share, so the choice becomes more varied.
It helps to have a picture. Suppose the model is completing "The weather today is" and four words are in the running. In our example they have scores of 3.0, 2.2, 1.0 and 0.0. The conversion is the same each time: raise e (about 2.72) to the power of each score divided by the temperature, then divide each result by the total so the four probabilities add to 100 per cent. You do not need to do this by hand, because the simulator below does it for you, but it is worth knowing there is no mystery in it. It is arithmetic that happens in a fraction of a second for every word.
Try the simulator. Move the temperature and watch the four bars. You can also change the scores themselves to see how the gaps between them decide how much temperature matters.
At a temperature of 1, word one has about 61 per cent, word two about 28 per cent, word three 8 per cent and word four 3 per cent. Drop to 0.3 and word one takes about 93 per cent. Raise to 2 and it falls to about 44 per cent, with the others sharing the rest. Same scores, different behaviour. That is all temperature does.
Here is another way to see the same picture using the interactive chart from the course. Move the slider below and watch the bars reshape. The same five words and the same scores are used each time; only the temperature changes.
Predict first: at a very low temperature, what happens to "chance two runs agree"? Decide, then check the slider.
It climbs towards 100 percent, because the top word is chosen almost every time. That is why a low setting gives steady, repeatable answers, and a high setting gives different answers on every run.
Why long answers drift apart
A single step is only part of the story. A reply is built one word after another, and each choice affects what comes next. If the model picks a slightly different word near the start, the rest of the sentence can go somewhere new. So even a small chance of a different word at every step adds up over a long answer.
Here is a simple way to see it. Suppose that at every step the best word has a 90 per cent chance of being chosen. The chance that a thirty-word answer picks the best word at every step is 0.9 multiplied by itself thirty times, which is only about 4 per cent. Even with a very strong favourite at each step, a long answer is almost certain to differ somewhere. The simulator below uses two words to show this in a simple form. It shows how the chance of choosing the best word at every step depends on temperature, the gap between the two scores and the length of the answer.
With a gap of 2, a temperature of 0.7 and thirty words, the chance is about 19 per cent: roughly one answer in five follows the best word the whole way. Raise the temperature to 1.5 and it falls close to zero. Lower it to 0.2 and it rises to nearly 100. This is why a low temperature feels "repeatable" and a high one feels "different every time". It is not that the model is sometimes right and sometimes wrong. It is that the dice are loaded more or less heavily at every step.
Choosing a setting
There is no best temperature, only a good fit for the task.
Extracting fields, classifying, answering from a document: use a low value, close to zero. You want the same, correct answer each time.
Everyday chat and summaries: a middle value works well.
Brainstorming names, ideas or phrasings: a higher value gives more variety, at the cost of more odd or weaker options.
The reasoning is the same in each case: match the setting to how much you want the answer to vary. If there is one right answer, variety is a risk, and you want the choice locked down. If there are many acceptable answers and you want to see a range, variety is the point.
The chart gives a rough sense of where people often start. These are starting points, not rules. The scale differs between providers, and some limit the range, so always check your provider's guidance and test on your own task.
The table shows the same thinking with the reasons and the risk on each side.
Task · Starting point · Why · What can go wrongExtracting fields from invoices · Very low · One right answer; consistency matters · A wrong value repeated every time
Classifying emails · Very low · Labels must be stable · A rare label is never chosen
Answering from a document · Low · You want the document's facts, not invention · Too rigid wording
Summaries · Low to middle · Faithful but readable · Dull or repetitive style
Everyday chat · Middle · Natural, some variety · Occasional odd phrasing
Names and slogans · Higher · Variety is the point · More weak or strange options
A very high temperature does not make a model more creative in a useful way; past a point it simply makes it more likely to pick poor words and wander off.
Related settings
Providers often offer other ways to limit the choice, such as keeping only the most likely handful of words (often called top-k) or only the smallest set of words whose probabilities add up to a chosen total (often called top-p). They narrow the list before the pick is made. Imagine the four words in our example. With a top-k of two, only the best two words stay in the running, and the others can never be picked, however high the temperature. With a top-p of 90 per cent, you keep the best words until their probabilities add up to 90 per cent, which here would be the first three.
These settings overlap with temperature in what they do, so changing both at once makes it hard to tell which caused a change. Change one control at a time, so you know which one made the difference. Many people simply set temperature and leave the others alone.
Why low does not mean identical
Setting the temperature to zero usually makes output close to repeatable, but not always exactly identical across runs or providers, so do not build a test that depends on byte-for-byte equality.
There are a few reasons. The computing hardware may do the same arithmetic in a slightly different order, producing tiny differences that occasionally tip a close choice. The provider may update the model behind the scenes. And two words with almost equal scores can swap places on a very small change. None of this is a bug in your code. It means that when you test, you should check the meaning of an answer, not whether it matches a saved copy character for character. For example, test that an extracted date is correct, not that the whole reply equals a stored string.
Testing a setting
How do you know which temperature suits your task? By testing, as with any other part of a prompt. A simple method works well.
1. Collect a set of real inputs, perhaps twenty to forty, with what a good answer looks like.
2. Pick two or three temperatures to compare, such as low, middle and high.
3. Run each input several times at each setting, say five, because a single run can mislead when answers vary.
4. Count how many answers are acceptable, and how many different answers you get for the same input.
5. Choose the lowest temperature that still gives answers you are happy with, unless variety is a goal.
The last point matters. For most business tasks, lower is safer: it costs nothing and removes a source of surprise. Raise it only when you want variety and have checked that the extra variety is useful.
When the answer is wrong, change the prompt first
A common instinct, when a model gives a bad answer, is to fiddle with temperature. This rarely helps, and it is worth understanding why. Temperature changes how the choice is made from the model's own scores. It does not change what the model knows or how it reads your instructions. If the model scores a wrong word highest, lowering the temperature makes it pick that wrong word more reliably. If it scores the right word highest but only just, a lower temperature can help, because it makes the favourite win more often.
So ask what kind of failure you are seeing.
The answer is wrong the same way every time. Temperature is not the cause. Look at the prompt, the information the model has, or whether the task is too hard for the model you chose.
The answer is sometimes right and sometimes wrong. The model's scores are close, and lowering the temperature may help. Check it on a test set, and also improve the prompt so the right answer is more clearly favoured.
The answer is right but the wording varies in annoying ways. This is the proper use of temperature, along with a firmer instruction about format.
The answers are dull and repetitive when you wanted variety. A higher temperature may help, but asking for several options, or giving different angles in the prompt, often works better.
A good habit is to treat temperature as one of the last things you tune, after the prompt, the information in the request and the model choice. Set it sensibly at the start, according to the task, and then spend your effort where it will make the most difference.
A practical way to set it
Here is a simple workflow that works for most projects.
1. Start from the task. If there is one right answer, begin close to zero. If you want variety, begin in the middle.
2. Fix everything else first. Get the prompt and the test set working at that setting.
3. Then test one step up and one step down. Run each version several times on the test set and compare the number of acceptable answers and the spread between runs.
4. Choose the lowest setting that works. Lower settings give fewer surprises.
5. Write it down. Record the setting and why you chose it, next to the prompt, so the next person does not change it without a reason.
6. Re-test when the model changes. A different model, or a provider update, can shift what a given setting does.
Explaining it to a colleague
You will probably have to explain temperature to people who are not engineers, such as a manager asking why the same question gives slightly different answers. A short, honest explanation helps. Something like: "The model gives each possible next word a score, and picks among them with a weighted dice roll. Temperature controls how loaded the dice are. Low means almost always the best word, so answers are steady. High means the weaker words get a real chance, so answers vary. For tasks with a single right answer we load the dice heavily, and for brainstorming we load them lightly."
Two further points are worth adding. First, small differences between runs are normal and are not a sign that something is broken. Second, if an answer needs to be exactly right every time, such as a figure in a report, the safeguard is not a low temperature on its own. It is a check on the output, such as comparing the figure with the source document, or showing a person the answer before it is used.
Three worked examples
Extracting invoice fields
A program reads supplier invoices and returns the invoice number, date and total. These are facts in the text, so there is one right answer for each. The team uses a temperature close to zero. On a test set, they run each invoice three times and find the answers identical every time, apart from one blurry scan. They add a rule to write "not found" when a field is unclear, which matters more than any temperature setting for that scan.
Naming a new product
A marketing team wants twenty possible names for a snack. Here variety is the point. They use a higher temperature and ask for ten names at a time, then run it a few times. Most names are ordinary, a few are odd, and a couple are good. They keep the good ones. At a very low temperature the same prompt gives nearly the same list every time, which is no use for brainstorming.
A support reply draft
A support tool drafts replies that an agent edits before sending. Too low a setting gives replies that sound stiff and identical, and too high a setting sometimes produces unusual wording. The team tries a middle value, reads fifty drafts and settles on it. They notice that for refund emails, where accuracy about policy matters most, they want a lower value, so they use a different setting for that type.
Common mistakes
Using one temperature for everything. Match the setting to the task.
Raising temperature to fix wrong answers. More randomness rarely corrects an error. Fix the prompt or add information instead.
Expecting zero to mean identical. Test for meaning, not exact text.
Judging a setting from one run. Run it several times and look at the spread.
Changing temperature and top-p together. You will not know which one helped.
Assuming the scale is the same everywhere. Providers differ, so check the documentation.
Using a high setting for facts. For extraction and answers from documents, keep it low.
Forgetting that long answers drift. Even a strong favourite at each step does not guarantee the same long answer.
Key takeaways
A model scores each next word, turns the scores into probabilities, then picks one.
Temperature divides the scores first: low makes the top word dominant, high spreads the choice.
Use low temperature for extraction and factual answers, higher for brainstorming.
Even a strong favourite at every step does not keep long answers identical, because small chances add up.
Top-k and top-p narrow the list of candidates, and are best changed one at a time.
Even at zero, outputs may not be perfectly identical, so test for meaning rather than exact text.
Test a setting on real examples, several runs each, and prefer the lowest one that works.
Check yourself
1. What does temperature do to the scores before they become probabilities?
It divides them, so a low temperature widens the gaps and a high one narrows them.
2. At a very low temperature, what is the chance the same top word is chosen on every run?
Very close to certain, which is why answers are steady and repeatable.
3. Why might a thirty-word answer differ from run to run even when the best word has a 90 per cent chance at each step?
The small chance of a different word at each step adds up over thirty steps.
4. You need to extract a date from an email. Which temperature do you choose, and why?
Very low, because there is one right answer and variety only adds risk.
5. Why should a test not require an answer to match a saved copy character for character, even at temperature zero?
Tiny differences in computation or model updates can change the wording, so check the meaning instead.
Try this
Set aside about forty-five minutes and pick one prompt you use, such as summarising notes or drafting a reply.
1. Write down what a good answer looks like, and whether you want one fixed answer or a range.
2. Collect five real inputs.
3. Run each input five times at a low setting and five times at a higher one. Write down how many different answers you got for each input.
4. Mark each answer as acceptable or not. Count the acceptable ones at each setting.
5. Use the first simulator with scores of your own invention to see how a gap between scores changes what temperature does.
6. Decide which setting you would use in a product, and write one sentence on why.
A good result is a small table of settings against acceptable answers and number of different answers, and one sentence saying which setting you chose and what would make you change it.
Tags: AI Engineering LLM basics Temperature Sampling Beginner