Lesson 6.1: Making results consistent from run to run
Learn why one prompt gives different answers, how to measure the variation, and the levers and regression checks that keep results steady.
By the end of this lesson you will be able to explain why one prompt can give different answers on different runs, measure how consistent a prompt is on a task, choose the levers that raise consistency without flattening everything that makes a model useful, and set up a simple regression check so that a change to a prompt or a model does not quietly change your results. You will see how the chance that two runs agree depends on how peaked the model's choice is, why five runs agreeing is much rarer than two, and how to run a before-and-after comparison on a set of cases. You should have read Lessons 3.1, 5.1 and 5.4, since consistency work uses the testing habits of the first two and the structured formats of the third. No programming is needed.
Lesson 2.4 explained that a model's answers can be wrong. This lesson deals with a quieter problem: answers that are not wrong but different. The same customer message is labelled "Billing" on Monday and "Account" on Tuesday. A summary has three bullets today and five tomorrow. A reply is warm in one run and curt in the next. For a person using an assistant for brainstorming, variety is a feature. For anyone who needs a process that behaves the same way each time, such as a team applying one standard, a pipeline feeding other software or a decision that must be explained later, it is a defect.
Why the same prompt gives different answers
Several things change from one run to the next, and it helps to separate them.
Source of variation · What happens · Can you control it?Sampling randomness · The model picks each next piece of text from a set of likely options, with some chance involved · Often, through a randomness setting, where your tool offers one
Ambiguity in the prompt · Several readings of your request are equally plausible, and different runs pick different ones · Yes, by sharpening the prompt
Differences in context · Another chat history, attachment or standing instruction changes what the model sees · Yes, by fixing the context
Model updates · The provider changes the model, and behaviour shifts · Partly, by pinning versions and testing
Input phrasing · Two inputs that mean the same thing are worded differently and get different treatment · Partly, by normalising inputs
The first deserves a short explanation. When a model writes, at each step it computes a spread of probabilities over possible next pieces of text and then chooses, with some randomness. A setting often called temperature controls how adventurous the choice is, as the temperature widget in the AI Engineering track shows. A low setting makes the model favour the most likely option almost every time. A high setting lets less likely options through, which gives variety and occasionally strange results. Many tools hide this setting, and some offer it in an advanced panel or through the developer interface.
A low setting reduces variation, but it does not guarantee identical output. Small differences in how the computation is run, ties between options and changes to the model can still produce different text. Treat "low randomness" as a strong nudge towards consistency, not as a lock.
Predict first: a prompt labels a customer message as one of three categories. Run it many times, and the model chooses "Billing" 80 per cent of the time, "Account" 15 per cent and "Other" 5 per cent. If you run it twice on the same message, how often will the two runs give the same label? Decide before reading on.
Many people say 80 per cent, since "Billing" is the usual answer. The right calculation counts every way the two runs can match. Both say Billing with probability 0.8 × 0.8 = 0.64. Both say Account with 0.15 × 0.15 = 0.0225. Both say Other with 0.05 × 0.05 = 0.0025. Adding these gives 0.665, so the two runs agree only about 67 per cent of the time, even though the model's single most likely answer is picked four times in five. Agreement between runs is always lower than the top choice's probability.
Measuring consistency
You cannot improve what you do not measure. A consistency test is simple.
1. Choose 20 to 30 inputs that represent the real work, including some awkward ones.
2. Run each input several times, at least three and ideally five, with the prompt and settings exactly the same each time, in separate fresh chats.
3. Record the outputs. For labels, tally the category per run. For text, score each run on your rubric, or compare key fields.
4. Compute the consistency. For each input, the share of runs that gave the most common answer, or the share of run pairs that agree. Average these across inputs.
5. Read the inconsistent cases. They are the most informative. Ask what about the input made the choice unstable.
Consistency is not correctness. A prompt can give the same wrong answer every time, which is consistent and useless, and a prompt can be right on average and unstable. You need both numbers, and they point at different fixes: low consistency suggests ambiguity or randomness, and low accuracy with high consistency suggests a flaw in the instructions or the examples.
At 80, 15 and 5 per cent, all five runs agree only 0.8⁵ + 0.15⁵ + 0.05⁵ = 0.328 + 0.0001 + 0 = about 33 per cent of the time. If a sharper prompt makes the top answer 95 per cent likely, with 4 and 1 per cent for the others, two runs agree 0.9025 + 0.0016 + 0.0001 = 90.4 per cent of the time and five agree 0.95⁵ = 77.4 per cent. A modest gain in how peaked the model's choice is brings a large gain in consistency, and the improvement grows with the number of runs.
The levers
Several levers raise consistency. They differ in strength and in what they cost.
Make the prompt less ambiguous
Most instability comes from a prompt that allows several readings. Everything from Lessons 1.3 and 1.4 applies: a clear task, defined terms, a fixed format and sharp constraints. A classification prompt that says "label the message as Billing, Fraud, Account or Other, and if two apply, choose the one that needs action first" removes a whole source of disagreement. When you read the inconsistent cases from your test, look for the rule you forgot to state.
Close the choices
Replace open answers with closed ones wherever you can. A fixed list of allowed labels, a scale with defined points, a yes or no with a stated threshold, or a template with named sections all narrow what the model may say. The structured outputs of Lesson 5.4 serve consistency as much as they serve software. A free-text field varies; an enumerated field does not.
Show examples
Examples, as in Lesson 2.1, pull the output towards a pattern. Choose them to cover the cases where runs disagree. If the same borderline message flips between two labels, add that message with the label you want.
Lower the randomness
Where your tool exposes a randomness setting, set it low for tasks that need steadiness, such as classification, extraction and formatting, and leave it higher for brainstorming and drafting where variety helps. Some tools also let you fix a seed, a number that starts the random choices from the same place each time, which makes runs more repeatable, though not perfectly. If your tool hides these settings, rely on the other levers.
Break the task into steps
A task with several judgements has several chances to vary, and the variations multiply. Splitting it, as in Lesson 3.2, lets you fix each judgement with its own tight prompt. If the labelling and the priority rating are separate steps with closed choices, each is far steadier than one prompt that asks for both and a paragraph of explanation.
Take the majority
Where each run is right more often than not and the answer can be compared, running several times and taking the most common result, as in Lesson 3.1, reduces the effect of any single odd run. It costs several calls, so use it for the cases that matter or for those flagged as uncertain.
Normalise the input
Different wordings of the same input can lead to different treatment. If you can, clean inputs before they reach the model: remove signatures and boilerplate, fix obvious typos, put dates and amounts in one format. A consistent input is the cheapest source of a consistent output.
Remember and reuse
For a process where the same input recurs, such as the same question asked by many people, the most consistent approach is to answer it once, check it and store the answer, so later requests get the stored version. That is consistency by design and not by luck, and it also saves cost.
Pin the version
A provider may update a model without changing its name in the tool you use. Where you can choose a specific, dated version, choose it for important processes, and re-test before you move to a new one, as Lesson 5.2 advised.
What you give up
Consistency has a price. A model forced into a narrow channel gives less variety, less surprise and sometimes less insight. For a creative task, a steady, safe answer may be a weak one. The right level depends on the job.
Task · Consistency matters · Variety mattersClassifying messages · Very high · None
Extracting fields · Very high · None
Applying a style sheet · High · Low
Drafting customer replies · Moderate: tone and facts, not wording · Some
Brainstorming names or ideas · Low · High
Writing marketing options · Low · High
Decide which column your task belongs to, and set the levers accordingly. A good practice for creative work is to ask for variety explicitly and then to choose: "Give me five different options, and tell me how they differ." That puts the variation where you can see it and pick.
Regression checks: when something changes
Consistency across runs is one half of the story. The other half is consistency across changes: a prompt edit, a new model version, a different tool setting. Any of them can alter outputs for cases you were not thinking about, and a change made to fix one problem can break another. A regression check compares the outputs before and after a change on the same set of inputs and shows you what moved.
The routine is short.
1. Keep a frozen test set, as in Lesson 5.1, with the outputs from the current prompt saved as the baseline.
2. Make the change. Change one thing.
3. Run the same inputs, with the same settings.
4. Compare case by case. Which outputs changed, and in which direction? For labels, count how many flipped. For text, compare key fields and scores.
5. Review every change. Some are improvements, some are harmless wording shifts and some are new errors. Do not accept a change on average results alone.
6. Decide and log. Record what changed, the numbers and your decision.
Suppose your 40-case set gave 36 correct labels with the old prompt. After an edit, 37 are correct. That sounds like a gain. But comparing the cases shows that 5 outputs changed: 3 flipped from wrong to right, and 2 from right to wrong. The net gain of one hides two new failures. If those two are cases of the kind you care most about, the edit may be a loss. Read the flips.
text
Compare these two sets of outputs for the same 40 inputs. List every input where
the output changed, with the old and new output side by side. For each change,
say whether the new output is better, worse or equivalent against the rubric
below, and why in one sentence. Do not summarise: list every change.
Rubric: [PASTE]
Old outputs: [PASTE]
New outputs: [PASTE]
You can ask an assistant to produce the comparison table, then check the rows that matter yourself. As with any grader, verify a sample of its judgements, as Lesson 5.1 advised.
Here is the arithmetic of a flip. If the old prompt got 34 of 40 right and the new one gets 36, the net gain is two. But if three outputs went from wrong to right and one went from right to wrong, four cases flipped, and one new failure is hiding inside the gain.
Worked examples
Example 1: a labelling prompt. A support team's classifier gives the same label on all five runs for only 41 per cent of its 30 test messages. Reading the unstable ones, they find that most involve two issues at once. They add a rule, "if a message has two issues, label the one that needs action first", plus an example, and fix the label list. All-five agreement rises to 80 per cent. Lowering the randomness setting takes it to 91. The remaining nine per cent are genuinely ambiguous messages, which they route to a person.
Example 2: a style problem. A team's replies vary in warmth from agent to agent because each uses their own prompt. They replace the personal prompts with one shared template with a style sheet and two golden examples, and check ten replies from each of four agents. The replies now differ in wording but not in tone or in the facts included, which is the level of consistency the team wants.
Example 3: a silent model update. A weekly report prompt starts producing four bullets in place of three, with no change on the team's side. Their frozen test set, run monthly, shows the format drift. They add an explicit "exactly three bullets" rule and a check, and set a reminder to re-run the set whenever the tool announces an update.
Common mistakes
Judging consistency from one run. Run several times, on several inputs.
Confusing consistent with correct. Measure both.
Relying on a randomness setting as a lock. It reduces variation and does not remove it.
Leaving choices open. Closed lists, scales and templates steady the output.
Forcing consistency on creative work. Ask for several options and choose.
Changing several things at once. You will not know which one moved the numbers.
Accepting a better average without reading the flips. New failures can hide inside a gain.
Never re-testing. Models and tools change, so schedule a check.
Key takeaways
Variation comes from sampling randomness, ambiguity, differing context, model updates and input phrasing. A low randomness setting helps but does not guarantee identical output.
Two runs agree less often than the top answer's probability: with an 80, 15 and 5 per cent split, only about 67 per cent, and all five runs agree about 33 per cent of the time.
Measure consistency by running 20 to 30 inputs several times and reading the unstable cases. Consistency and correctness are different, so measure both.
The levers are: a sharper prompt, closed choices, examples, lower randomness, task splitting, majority voting, input cleaning, stored answers and pinned versions.
Consistency costs variety, so match the setting to the task: high for labelling and extraction, low for brainstorming.
A regression check compares outputs before and after any change on a frozen set. Read every flip, since a net gain can hide new failures.
Check yourself
1. If a model gives answer A 70 per cent of the time and answer B 30 per cent, how often do two runs agree?
0.7 × 0.7 + 0.3 × 0.3 = 0.49 + 0.09 = 0.58, or 58 per cent.
2. Why can a prompt be consistent and still bad?
It can give the same wrong answer every time. Consistency measures steadiness, not correctness.
3. Name three ways to make a prompt's output steadier.
Sharpen ambiguous instructions, close the choices with fixed lists or templates, add examples, lower the randomness, split the task, take a majority or pin the version.
4. After an edit, the number correct goes from 34 to 36 out of 40, but 4 outputs flipped. What should you do?
Read the four flips. Some may have gone from right to wrong, so the net gain of two may hide new failures.
5. Why might you not want maximum consistency when brainstorming?
You want variety and surprise, and a narrow, steady output gives fewer ideas.
Try this
You need an assistant and about fifty minutes.
1. Choose a prompt you use for a repeatable job, such as labelling, extracting or applying a style. Write down the inputs: 15 to 20 realistic ones, a few awkward.
2. Run each input three times in separate fresh chats, with the same prompt. Record the outputs.
3. For each input, mark whether all three runs agreed. Compute the share that did. Read the unstable ones and write down why each is unstable.
4. Fix the prompt using two levers, such as a closed list of labels and one example for the unstable case. Rerun the unstable cases three times each and recompute.
5. Use the agreement simulator with your own probabilities to see what moving from an 80 per cent to a 95 per cent top choice would do for five runs.
6. Make a deliberate change to the prompt, such as rewording one rule. Run the full set, compare with your baseline case by case, and list the flips. Decide whether to keep the change.
7. Fill in this table.
Item · Before · AfterInputs where all runs agreed · ·
Number correct against your own answers · ·
Flips caused by the later edit · ·
Decision · ·
A good result is a measured consistency figure for your prompt, a list of the unstable cases with their causes, a version that is measurably steadier and a saved baseline for future regression checks. In the next lesson we turn to a different kind of reliability: what happens when the text a model reads is trying to give it orders.
Tags: Prompt Engineering Consistency Reliability Randomness Advanced