Lesson 5.1: Testing and comparing prompts with rubrics and test sets
Learn to build a small test set, score outputs with a rubric, use a model as a grader carefully, and tell whether one prompt really beats another.
By the end of this lesson you will be able to build a test set that reflects the work your prompt will really do, write a rubric that two people would score the same way, use a model as a grader without being fooled by its habits, and decide with a little arithmetic whether one prompt is genuinely better than another or only lucky on a few cases. You will see why a handful of tests cannot separate two prompts that differ by ten points, how many cases you would need, and how to read a win rate with its margin of error. You should have read Lessons 2.1, 3.4 and 4.2, since testing builds on the sampling arithmetic of the first, the comparison habits of the second and the pilot-and-sample approach of the third. No programming is required, though builders will find the same ideas apply when the process is automated.
Until now this course has asked you to judge outputs by reading them. That works for one email. It fails when a prompt will run a thousand times, when several people share it, or when you are choosing between two versions. At that point intuition is not enough, because a few good-looking outputs prove very little, and a prompt that wins on the cases you happened to try may lose on the next ones. Testing replaces the impression with evidence. This is the gap between using prompts and engineering them.
Why look at outputs is not a test
Suppose you rewrite a prompt and try it on three inputs. The first output is better, the second the same, and the third slightly worse. What have you learned? Very little. Outputs vary from run to run, three cases are a tiny sample, and you have no standard to say what better means. A test has four ingredients, and each one fixes a weakness of casual checking.
Ingredient · What it fixesA test set of realistic cases · You try the prompt on more than the cases you thought of
Expected results or a rubric · You judge by a standard, not a mood
A scoring method · Everyone scores the same way, and the scores can be compared
A comparison rule · You know when a difference is large enough to believe
The remainder of the lesson takes these in turn.
Building a test set
A test set is a collection of inputs, with the correct or acceptable outputs where they exist, that you run every version of a prompt against. Its value depends on how well it represents real use. A good set has four kinds of case.
Typical cases, the everyday inputs that make up most of the work. They tell you whether the prompt does its main job.
Edge cases, unusual but legitimate inputs: very short, very long, two issues in one message, mixed languages, missing information.
Past failures, every input that ever produced a bad result. These are gold, because each one is a mistake you have already paid for. Add them as you find them, and they become a permanent guard against regressions.
Adversarial cases, inputs designed to break the prompt: contradictory instructions, a request outside its scope, text that tries to override your instructions. Lesson 6.2 covers these.
For a prompt that classifies support messages, a set of 40 cases might hold 20 typical, 10 edge, 6 past failures and 4 adversarial.
Where do the cases come from? Real data is best: past messages, documents or requests, with sensitive details removed. If you have none, ask a colleague who does the job to write fifteen realistic inputs, and ask an assistant for more, as in Lesson 3.4, then check them. Whichever way you source them, keep the expected output for each case wherever a definite answer exists, such as a category, a number or a yes or no. Where outputs are open-ended, such as a summary, you will use a rubric instead.
A set does not need to be large to start. Twenty well-chosen cases will show up most large problems. It does need to be kept apart from your examples: the cases you show in the prompt, as in Lesson 2.1, must not appear in the test set, or the prompt will look better than it is.
Writing a rubric
When there is no single right answer, a rubric defines what good looks like. It is a short list of criteria, each with a scale and a description of each score, so that two people reading the same output would give nearly the same marks.
A weak criterion is "quality: 1 to 5". It means a different thing to every scorer. A strong one is specific.
Criterion · 1 (fails) · 3 (acceptable) · 5 (excellent)Accuracy · Contains a claim not in the source · All claims in the source, one minor slip · Every claim traceable to the source
Completeness · Misses two or more key points · Covers the main points · Covers all key points, in the right order
Length and format · Ignores the limit or the format · Close to the limit · Within the limit and in the requested format
Tone · Wrong register for the reader · Mostly right · Matches the style sheet throughout
Three habits make rubrics reliable. Write anchors: a sentence for at least the lowest, middle and top score. Use few criteria, four or five, since more makes scoring slow and noisy. And separate dealbreakers from degrees: a made-up fact might be an automatic fail whatever the other scores, which you can express as a pass or fail line on its own. To check that a rubric works, have two people score the same ten outputs independently and compare. If they disagree often, the descriptions need sharpening.
Predict first: you have two prompts and a 20-case test set. Prompt B beats Prompt A on 14 cases, loses on 6 and no case is tied. B wins 70 per cent of the time. How sure can you be that B is genuinely better? Decide before reading on.
Less sure than 70 per cent suggests. With only 20 decisive cases, the plausible range for B's true win rate is wide. The margin of error is 1.96 × √(0.7 × 0.3 ÷ 20) = 0.20, so the range runs from about 50 to 90 per cent. The bottom of that range is a coin flip, which means the data cannot rule out that the two prompts are equal. Fourteen wins to six looks convincing, and it is not enough on its own.
Comparing two prompts honestly
The cleanest way to compare is side by side on the same cases, with the same rubric, and ideally scored without knowing which prompt produced which output. For each case record a win for A, a win for B or a tie. The share of wins, among cases that were not tied, is the win rate. Then ask how much it could be due to chance.
Try 28 wins to 12 losses: the rate is still 70 per cent, but with 40 decisive cases the margin falls to about 14 points and the lowest plausible rate is about 56 per cent, so the evidence now favours the new prompt. The same win rate means more when it rests on more cases.
Measuring accuracy when there is a right answer
Where each case has a correct answer, you can compare the accuracy of two prompts instead. Suppose Prompt A scores 70 per cent and you hope Prompt B scores 80. How many test cases would you need to tell them apart reliably, with 5 per cent significance and 80 per cent power? The standard calculation from the Data Science course gives n per prompt = 7.84 × (0.7 × 0.3 + 0.8 × 0.2) ÷ (0.1)², which is 7.84 × 0.37 ÷ 0.01 = 290 cases per prompt. A test of 20 cases has no hope of detecting a ten-point gain. A gain of twenty points, 70 to 90 per cent, needs 7.84 × (0.21 + 0.09) ÷ 0.04 = 59 cases.
These figures are sobering. Most teams cannot assemble 300 labelled cases for every comparison. The practical response has three parts. Aim for large differences: a small tweak that moves accuracy by two points is not worth chasing, and a structural change that moves it by twenty will show on 60 cases. Reuse the cases: the same set serves every version, so building it once pays back repeatedly. And be honest about uncertainty: report "B won 16 of 22 decisive cases, a lean in its favour" and not "B is better".
Noise in the model itself
Outputs vary from run to run, so a single run on a case may flatter or punish a prompt. For important comparisons, run each case two or three times and count a case as passed only if it passes consistently, or use the average score. If a prompt passes a case one time in three, that is a weakness you want to see. Setting the model to be less random, where your tool allows, reduces the variation, a topic Lesson 6.1 covers.
Using a model as the grader
Scoring 40 cases by hand for every prompt version is slow. A common shortcut is to have a model score the outputs against your rubric, an approach often called model grading. It can save a great deal of time, and it has known weaknesses, so use it with care.
It favours longer, more elaborate answers, even when they are no better. Include length in the rubric explicitly, and check whether scores rise with length alone.
It can favour whichever answer comes first, a position bias. When comparing two outputs, run the comparison twice with the order swapped, and count a win only if it holds both ways.
It can prefer outputs that resemble its own style, so a grader from the same family as the writer may be lenient.
It cannot check facts it does not have. Give it the source and ask for a quotation supporting each claim.
Its scores drift with its wording, so keep the grading prompt fixed and versioned like any other.
The remedy is to calibrate the grader against people. Take 20 outputs, score them yourself, and let the grader score them with the same rubric. Count how often you agree. If you match on 17 of 20, that is 85 per cent agreement, and the margin on that figure is wide, but it tells you the grader is usable for rough comparisons. If you match on 11, it is not, and you should sharpen the rubric or score by hand. Keep checking: sample a few graded cases each round and confirm that you still agree with the grader.
text
You are grading a customer reply against a rubric. You will receive the customer
message, the facts the agent had, and the reply.
For each criterion, give a score from 1 to 5 using the descriptions below and
quote the words in the reply that justify the score. Do not reward length.
If the reply contains any claim not in the facts, set Accuracy to 1 and say which.
Rubric:
[PASTE THE FOUR-CRITERION RUBRIC]
Customer message: """[PASTE]"""
Facts: """[PASTE]"""
Reply: """[PASTE]"""
Output as a table with the columns Criterion, Score and Evidence.
A testing routine you can keep
Put the pieces into a short routine, and use it each time you change an important prompt.
1. Freeze the test set and the rubric. Note their versions.
2. Run the current prompt, the baseline, on every case. Record the scores.
3. Change one thing, and run the new version on the same cases.
4. Compare case by case: wins, losses, ties. Read the cases that changed, since a gain on average can hide a new failure.
5. Check the margin, using the win-rate or sample-size arithmetic.
6. Add any new failure to the test set.
7. Log the result: the date, the change, the scores and the decision.
The log doubles as a record for anyone who asks why the prompt reads as it does. After a few rounds you will know which kinds of change move the numbers, which is how experienced prompt engineers develop their judgement.
Worked examples
Example 1: a classification prompt. A support team has a labelling prompt at 75 per cent accuracy on a 40-case set. They add three examples covering the borderline cases and raise it to 88 per cent, which is 35 of 40. Is the gain real? The sample-size arithmetic says detecting a 13-point gain needs roughly 150 cases per prompt, so on 40 cases the result is only suggestive. They read the five cases that flipped, see that all five are the borderline types the examples addressed, and judge the change real, while queuing 60 more cases to confirm.
Example 2: a summary rubric. A team's summaries are scored on accuracy, completeness, length and tone. Two scorers rate ten outputs independently and agree within one point on 34 of 40 scores, which they judge acceptable. They then calibrate a model grader on the same ten outputs, and it agrees on 31 of 40. Good enough for screening, so they use it for weekly checks and hand-score a random five each time.
Example 3: a regression caught. A teammate edits a shared prompt to make replies friendlier. The overall rubric score rises, but the past-failure cases show two regressions: the new prompt promises a refund in two cases where the facts did not. The test set caught what an average would have hidden, and the edit was revised. This is the main reason to keep failures in the set.
Common mistakes
Judging on three cases. Use a real test set, and report results with their uncertainty.
A set of only typical cases. Include edge cases, past failures and adversarial inputs.
Test cases that appear in the prompt as examples. Keep them apart.
Vague rubrics. Use specific criteria with anchors, and check that two scorers agree.
Believing a small win. Fourteen to six on twenty cases is not conclusive. Do the arithmetic.
Trusting a model grader blindly. Watch for length and position bias, and calibrate against people.
Changing several things at once. You will not know which change worked.
Not recording failures. Every new failure belongs in the set.
Key takeaways
A test needs a test set, a rubric or expected answers, a scoring method and a rule for when a difference counts.
A good set mixes typical, edge, past-failure and adversarial cases, comes from real data, and stays separate from your prompt's examples.
Rubrics work when criteria are few and specific, scores have anchors, and two people agree on the same outputs. Treat made-up facts as a dealbreaker.
A 14 to 6 win on 20 cases gives a plausible win rate of about 50 to 90 per cent. Detecting a ten-point accuracy gain needs roughly 290 cases per prompt, and a twenty-point gain about 59.
A model grader saves time and is biased towards length, order and its own style. Run comparisons both ways round and calibrate against human scores.
Change one thing at a time, read the cases that flipped, add new failures to the set and log every result.
Check yourself
1. Name the four kinds of case in a good test set.
Typical cases, edge cases, past failures and adversarial cases.
2. A new prompt wins 9 and loses 3 of 12 decisive cases. Is that conclusive?
No. The win rate is 75 per cent, but with 12 cases the margin is about 24 points, so the lowest plausible rate is about 51 per cent. It is suggestive only.
3. Why must test cases be kept separate from prompt examples?
If the prompt has seen the answers, it will look better than it is on new inputs.
4. What are two known weaknesses of model graders?
A bias towards longer answers, a bias towards the first option, a preference for its own style, and inability to check facts not provided.
5. About how many cases per prompt do you need to detect a rise from 60 to 80 per cent?
7.84 × (0.24 + 0.16) ÷ 0.04 = about 79 cases per prompt.
Try this
You need an assistant and about sixty minutes.
1. Pick a prompt you rely on, such as a classifier, a summary or a reply template. Write the task in one line and decide what a pass looks like.
2. Build a test set of 20 to 30 cases: about half typical, a quarter edge, a few past failures and two adversarial. Remove sensitive details. Keep expected answers where they exist.
3. Write a rubric with four criteria and a short description for scores 1, 3 and 5. Have a colleague score five outputs independently, and compare.
4. Run the current prompt on every case, and score it. This is your baseline.
5. Change one thing, such as adding examples or a fallback line, and rerun on the same cases. Record wins, losses and ties.
6. Use the win-rate simulator and the sample-size simulator to say how sure you can be, and how many cases a firm answer would need.
7. Fill in this table.
Item · Baseline · New versionCases passed · ·
Mean rubric score · ·
Cases that flipped · ·
Margin of error · ·
Decision · ·
A good result is a reusable test set and rubric, an honest comparison with its uncertainty stated, and a one-line log entry for the change. In the next lesson we apply the same discipline to a decision that has large consequences: which model to use.
Tags: Prompt Engineering Evaluation Test sets Rubrics Advanced