Lesson 5.2: Choosing and comparing models for your task
Learn to compare models on your own task by quality, cost, speed and risk, work out cost per outcome, and route easy work to cheaper models.
By the end of this lesson you will be able to list the factors that decide which model suits a task, compare candidates on your own test set instead of on published rankings, work out the true cost of a model once the price of errors is counted, decide when routing easy requests to a cheaper model pays, and plan for the day a model changes or disappears. You will see how a model that costs four times as much per call can be the cheaper choice overall, and where the break-even point lies. You should have read Lesson 5.1, since every comparison here rests on a test set and a rubric, and Lesson 1.2, since context windows and token costs matter here. No programming is needed, and this lesson deliberately names no particular products, because the market changes faster than a course can.
For most people the choice of model is made for them: it is whatever the company's tool provides. But as soon as you build a process around an assistant, or influence which tool a team adopts, the question becomes real, and a bad answer is expensive in either direction. Pay too much and a simple task burns budget. Pay too little and errors cost more than the saving. The way to decide is to measure on your own work.
What "a model" means in practice
A model is a trained system that reads a prompt and writes a reply. Providers usually offer a family of them, and a few ideas help you read the menu without needing names.
Tiers. Within a family there are typically smaller, faster, cheaper models and larger, slower, costlier ones. The larger ones tend to do better on hard reasoning, long instructions and subtle writing, and the smaller ones are often enough for routine sorting, extraction and short replies.
Versions. Each model gets updated or replaced over time, and behaviour can change between versions, sometimes in ways that break a prompt that used to work.
Modes. Some offer a mode that reasons at length before answering, as Lesson 3.1 described, which costs more time and money.
Hosting. A model may be used through a provider's service, where your text travels to their servers, or run on infrastructure you control, which changes privacy and cost in ways covered below.
No model is best at everything. The right question is not "which is the best model?" but "which model is good enough on my task at the lowest total cost and risk?"
The factors that matter
Factor · The question to ask · Why it mattersQuality on your task · How does it score on my test set and rubric? · The one factor that is entirely yours to measure
Cost · What does a call cost, and what does an error cost? · Price per call is only part of the cost
Speed · How long until the first words, and until the full reply? · A customer waiting in a chat needs a fast answer
Context window · Can it hold my longest document and instructions? · A window that is too small makes the task impossible
Features · Does it support images, tools or structured output if I need them? · A missing feature forces workarounds
Languages · How well does it handle the languages my users write in? · Quality can differ sharply outside English
Data handling · Where does the data go, how long is it kept, who can see it? · Legal and trust obligations
Reliability · Is the service stable, with clear limits and notice of changes? · A process cannot depend on an unreliable link
Control · Can I fix a version, and is there a fallback? · Prevents surprise breakage
Rank the factors for your task before you look at any model. A real-time chat feature will put speed near the top. A document-review task will put quality and context length there. A process touching customer financial data will put data handling first and may rule some options out entirely.
Published rankings are not your test
Providers and independent groups publish benchmark scores, and they are useful for a rough sense of capability. They are a poor guide to your task, for three reasons. The tasks in a benchmark differ from yours, so a model that leads on maths puzzles may not lead on drafting replies in your house voice. Benchmarks can be leaked or targeted, since models may have seen the questions, or been tuned to do well on them. And the prompt matters: a benchmark uses one fixed prompt style, and your best prompt for each model may differ. Treat a leaderboard as a way to pick two or three candidates, and then decide on your own data.
Predict first: two models are tested on the same 40 cases. The cheaper one scores 82 per cent and costs ₹0.50 a call. The dearer one scores 93 per cent and costs ₹2.00 a call. Each wrong answer costs ₹30 to fix. Which is cheaper overall? Decide before reading on.
Most people choose the cheaper model, because ₹2.00 is four times ₹0.50. Count the errors, though. The cheaper model is wrong 18 per cent of the time, so each call carries an expected fixing cost of 0.18 × 30 = ₹5.40, making the total cost per call ₹5.90. The dearer model is wrong 7 per cent of the time, an expected fixing cost of 0.07 × 30 = ₹2.10, for a total of ₹4.10. The model that costs four times as much per call is about 30 per cent cheaper once errors are counted.
The result depends on the price of an error, which is why that number is the most important input. If an error costs only ₹5, because it is caught instantly and costs nothing, the ranking reverses: the mid-size model costs 0.50 + 0.18 × 5 = ₹1.40 and the large one 2.00 + 0.07 × 5 = ₹2.35. The break-even error cost, where the two cost the same, is (2.00 − 0.50) ÷ (0.18 − 0.07) = ₹13.64. Above that, the larger model wins. Below it, the smaller one does.
Two cautions follow. The accuracy figures are estimates from your test set, and a gap of eleven points on 40 cases is uncertain, as Lesson 5.1 showed. And errors are not all alike: some are caught at once and cost minutes, while others reach a customer and cost far more. Weight them by what they cost, not only by how many there are.
A fair comparison protocol
A comparison is only as good as its method. A sound one has six parts.
1. Fix the task and the rubric before you test anything, and freeze the test set.
2. Pick two to four candidates from different tiers, using rankings only to shortlist.
3. Adapt the prompt for each model. A prompt tuned on one model is not automatically best for another, and giving each a fair tuning effort is part of fairness. At minimum, try the same instructions, and one variant per model.
4. Run every case on every model, two or three times where outputs vary, and score with the same method.
5. Record cost and speed, not just quality: tokens used, price, time to first words and total time.
6. Compute the cost per outcome, including the cost of errors, and note the uncertainty.
Resist the temptation to test on the cases where you expect a difference. Use the same representative set for all, including the edge cases, since a cheaper model often fails on exactly those.
Speed and the user's patience
Quality and cost are not the whole story. Speed shapes how a tool feels. Two measures matter: how long before the first words appear, and how long until the reply is complete. For a chat with a customer, a reply that starts within a second or two feels responsive even if it takes ten seconds to finish, because the person can start reading. For a batch job overnight, speed is irrelevant and cost dominates. Larger models and reasoning modes are usually slower, so a quality gain often comes with a wait. Ask what delay your users will tolerate, and test against that.
Routing: use the cheap model for easy work
If most requests are simple and a minority are hard, using the largest model for all of them wastes money. Routing sends each request to the cheapest model that can handle it. Easy cases go to a small model, and hard ones to a large one.
Suppose 70 per cent of requests are easy. A small model handles them at ₹0.10 a call and is right 95 per cent of the time. The large model costs ₹2.00, is right 98 per cent on easy cases and 85 per cent on hard ones. With an error costing ₹30, using the large model for everything costs 2.00 + (0.7 × 0.02 + 0.3 × 0.15) × 30 = 2.00 + 0.059 × 30 = ₹3.77 per call. Routing easy cases to the small model and hard ones to the large costs 0.7 × (0.10 + 0.05 × 30) + 0.3 × (2.00 + 0.15 × 30) = 0.7 × 1.60 + 0.3 × 6.50 = 1.12 + 1.95 = ₹3.07 per call. That is about 19 per cent cheaper.
How do you tell easy from hard? Simple rules sometimes suffice, such as message length or a keyword. A small model can do the sorting. Or you can try the small model first and escalate when it expresses low confidence or its answer fails a check. Each method has costs, and the savings in the example vanish if sorting is poor. Test the whole routed system on your test set, not only its parts.
Data handling and risk
Where your text goes matters as much as how well the model answers. Before you send anything real, find out the answers to a handful of questions.
Where is the data processed and stored, and in which country?
Is it kept, and for how long? Is it used to train future models, and can you opt out?
Who at the provider can see it?
What certifications or contractual terms apply, and do they meet your organisation's rules and the data protection law that governs your customers' information?
Can you delete it on request?
For sensitive material, options include a tier with stronger data terms, a deployment inside your own cloud account, or a model that runs on your own hardware, which keeps data in-house at the price of running it yourself and usually a lower quality. These are decisions for your security and legal teams, not for an individual user, and Lesson 6.3 returns to the subject. The point here is that a model that wins on quality but fails on data handling is not an option.
Planning for change
Models change. A version may be retired, a price may rise, behaviour may shift after an update, or a service may be unavailable for an afternoon. Protect yourself with a few habits.
Pin the version where you can, so that behaviour does not change without your knowing.
Re-run your test set whenever the version changes, and on a regular schedule, say quarterly, since drift can be quiet.
Keep a fallback, a second model or a manual path, for when the first fails or becomes too expensive.
Keep prompts and test sets in your own files, not only inside one vendor's tool, so that moving is possible.
Watch the announcements for retirement dates, and leave time to re-test.
Avoid features that lock you in unless the benefit is clear.
A change is also an opportunity. Each new generation tends to improve at the same price or lower, so a quarterly re-test may reveal that a cheaper model now does the job.
Worked examples
Example 1: a support classifier. A team tests three models on 120 labelled messages. The small one is right 86 per cent of the time, the mid-size 91 and the large 93. Each wrong label costs about ₹4 of agent time. With that error cost, the small model's total is 0.10 + 0.14 × 4 = ₹0.66 per message and the large one's is 2.00 + 0.07 × 4 = ₹2.28, so the small model wins by a wide margin. They choose it and queue a review of the 14 per cent errors to see whether examples could lift it.
Example 2: a contract summary tool. For a task where errors can reach a client and cost thousands of rupees in rework and goodwill, the same arithmetic reverses. A model that is right 95 per cent of the time against 85 per cent saves more in avoided errors than it costs in extra tokens, even at several times the price. They choose the larger model, and keep a human check on every output.
Example 3: a language gap. A bank tests a model on replies in English and Hindi. Accuracy is 92 per cent in English and 78 in Hindi, a gap the published benchmarks had not shown. They test a second model, which scores 90 and 88, and choose it, though it is slightly worse in English. Without a test in the language their customers use, they would have shipped the weaker experience.
Common mistakes
Choosing on price per call. Count the cost of errors.
Choosing on a leaderboard. Shortlist with it and decide on your own test set.
Using one prompt for every model. Give each a fair adaptation.
Testing only in English. Test the languages your users write in.
Ignoring speed. A slow reply can ruin a chat experience.
Overlooking data handling. A model that fails your privacy rules is out, whatever its quality.
Never re-testing. Versions change, and a cheaper model may now be enough.
No fallback. Plan for outages, price changes and retirements.
Key takeaways
No model is best at everything. Choose the one that is good enough on your task at the lowest total cost and risk.
Rank the factors first: quality on your task, cost, speed, context window, features, languages, data handling, reliability and control.
Use public rankings to shortlist and your own test set to decide, with a prompt adapted to each model.
Cost per call is not cost per outcome. With an error costing ₹30, a model at ₹2.00 and 93 per cent beats one at ₹0.50 and 82 per cent, with break-even at about ₹13.64 per error.
Routing easy requests to a small model can cut cost, in the example from ₹3.77 to ₹3.07 per call, if the sorting is cheap and accurate.
Check data handling before sending real data, pin versions, re-test regularly and keep a fallback.
Check yourself
1. Why is the cheapest model per call not always the cheapest overall?
Errors cost money to find and fix. A cheaper but less accurate model can cost more once the cost of its errors is counted.
2. Model A costs ₹1 and is right 90 per cent of the time, Model B costs ₹3 and is right 97 per cent. An error costs ₹20. Which is cheaper per call?
A: 1 + 0.10 × 20 = ₹3.00. B: 3 + 0.03 × 20 = ₹3.60. A is cheaper.
3. At what error cost are the two models in question 2 equal?
(3 − 1) ÷ (0.10 − 0.03) = ₹28.57.
4. Name two reasons published benchmarks may mislead on your task.
The tasks differ from yours, models may have seen the questions, and a fixed prompt may not suit each model.
5. Why re-run your test set when a model version changes?
Behaviour can shift between versions, and a prompt that worked may fail, or a cheaper model may now be enough.
Try this
You need access to at least two models or tiers and about sixty minutes. If you have only one tool, do the arithmetic exercises and plan the test.
1. Choose a task from your work and define the test set and rubric from Lesson 5.1. Aim for at least 20 cases, in each language you use.
2. Estimate the cost of one error: the time to find and fix it, plus any harm if it reaches a customer. Write down a low and a high estimate.
3. Run the same prompt on two or three models or tiers, then a lightly adapted prompt on each. Record the score, the tokens used, the price and the time to a full reply.
4. Use the cost simulator with your accuracy and cost figures at both the low and the high error cost. Note whether the winner changes, and the break-even error cost.
5. Decide whether a routing split would help, by estimating the share of easy requests and testing a small model on only those.
6. List your data-handling requirements and mark each candidate as meeting them or not.
7. Fill in this table.
Item · Model 1 · Model 2 · Model 3Accuracy on the test set · · ·
Cost per call · · ·
Time to full reply · · ·
Total cost per call at your error cost · · ·
Meets data requirements? · · ·
A good result is a choice backed by your own numbers, a stated error cost and break-even point, a fallback and a date to re-test. In the next lesson we turn to a task where models most often fail quietly: working from long documents and showing where each claim comes from.
Tags: Prompt Engineering Model selection Cost Evaluation Advanced