Lesson 3.1: Designing an agent from a job description
Learn to design an agent the way you would define a job: duties, boundaries, tools, success measure and test cases, with the numbers that show whether it pays.
By the end of this lesson you will be able to turn a vague wish for an agent into a one-page job description, break the job into tasks and decide for each whether a model, a tool, plain code or a person should do it, write the test cases that define success before building anything, and work out from the resolve rate whether the agent saves money. You will price an invoice-query agent that resolves 85 per cent of queries at ₹4 each and passes the rest to a person, find that it brings the average cost per query from ₹30 to ₹8.50, and discover that it breaks even at a resolve rate of only 13.3 per cent. You will also see why a test set of 50 cases leaves a margin of about 10 points around a measured 85 per cent. You should have read Lessons 1.3 and 1.4, which cover levels of autonomy and the parts of an agent. The mathematics is a weighted average and a square root.
The first two modules gave you the concepts and the tools. This module is about building. The most common mistake in building is starting too early: opening an editor and wiring a model to a few tools before anyone has said what the agent is for. The cure is to design the agent as you would define a role for a new colleague. Write the job down, list its duties, set its boundaries, say how its work will be judged and prepare the first week of work to test it on. Only then build. This lesson teaches that design step, and the next three lessons build what it describes.
Start with a job description
A job description for a person says who they report to, what they do, what they may decide alone and what they must refer, which tools they use and how their performance is measured. An agent deserves the same, written in the same plain language. It is also a discipline: a wish that cannot be written as a job description is not yet ready to be built.
Take a running example. An accounts team receives about 300 questions a day from suppliers about invoices, mostly "why was my invoice rejected?" and "has it been paid?". A staff member handles each in about 6 minutes, looking in two systems and replying. The team wants an agent to take the routine ones.
A good one-page job description has eight parts.
Part · Question it answers · ExamplePurpose · What is this agent for, in one sentence? · Answer suppliers' questions about the status of their invoices and the reasons for rejection
Inputs · What arrives, from where, in what form? · A supplier's email or portal message, with a supplier ID
Outputs · What must come out, in what form? · A reply to the supplier, and a note in the case record
Duties · What tasks make up the job? · Identify the invoice, look up its status, explain a rejection, tell the supplier what to fix
Authority · What may it do alone, and what needs a person? · Read invoices and rules, reply to suppliers. May not change any invoice, promise a payment date or discuss another supplier
Tools and data · What does it need access to? · Invoice system (read), rejection rules (read), reply tool, escalate tool
Success measure · How will we know it does the job well? · Correct and complete replies for at least 85 per cent of queries, no wrong disclosures, an average handling time under 2 minutes
Escalation · When does it hand over, and with what? · Unknown rejection code, invoice not found, an angry or legal message, anything outside its authority. Hands over the case with a summary
Each row removes a decision that would otherwise be made by accident. The authority row, for instance, settles the level of autonomy from Lesson 1.3 before a line of code exists. The success measure row forces a statement of what "good" means, which is the thing most projects lack and then argue about later.
Write the description in prose and bullet points that a non-technical manager can check. If the manager reads "may not promise a payment date" and objects, you have found a disagreement while it is still cheap to fix.
Break the job into tasks
The duties in a job description are not all the same kind of work. Take each one and ask who or what should perform it. There are four candidates.
Performer · Best for · Example from the invoice jobPlain code · Steps that are fixed and exact · Check that an invoice number has the right format; calculate the days since submission
A tool · Fetching facts or acting in a system · Look up the invoice; send the reply
The model · Understanding messy language and judging · Work out which invoice a supplier means from a rambling email; explain a rejection in friendly words
A person · High stakes, authority, empathy or what the agent cannot do · Disputed amounts, legal threats, exceptions to policy
Applying this to the duties gives a task map.
Duty · Performer · ReasonFind the invoice number in the message · Model · Messages are free text and vary
Check the number's format · Code · Exact, cheap, never wrong
Look up the invoice · Tool · A fact held in a system
Look up the rejection rule · Tool · A fact held in a system
Explain the rejection in plain words · Model · Needs good writing
Say what to change · Model, using the rule text · Needs judgement, but is grounded in the rule
Send the reply · Tool · An action in the world, with the limit enforced in code
Handle a disputed amount · Person · Needs authority
The map often shrinks the part that is truly "agentic". Only the first, fifth and sixth rows need the model, and even the order of steps here is nearly fixed: find, look up, look up, explain, send. This is an important finding. If the sequence barely varies, a workflow in the sense of Lesson 1.1 will be cheaper, faster and easier to test than a free-roaming agent, with the model used for the language steps. An agent earns its place where the path varies: some rejections need one more lookup, some need a question to the supplier, some need escalating at once. Writing the task map tells you honestly how much freedom the job needs, and you can give the model exactly that much and no more.
Predict first: in the invoice job, most queries follow the same path but a minority need extra steps. A team can build it as a fixed workflow with a fallback to a person, or as a free agent that can choose any tool. Which is more likely to be the better first version? Decide before reading on.
The workflow with a fallback, in most cases. It is predictable, testable and cheap. Because the unusual cases escalate to a person rather than being handled in clever ways, the first version is safe, and the log of what was escalated shows precisely which extra abilities would be worth adding. Many successful agents began as workflows and gained freedom one step at a time, as the evidence supported it. Starting with maximum freedom is the harder and riskier road.
Decide how success is measured, before building
The success measure row deserves its own attention, because it decides whether you will ever know if the agent works. Good measures share three properties.
They are about the outcome, not the activity. "Replies correct and complete" is an outcome. "Number of tool calls" is an activity.
They can be checked. Someone, or something, can look at a case and say whether it passed. A reply can be compared with the true reason from the system.
They have a threshold agreed in advance. "At least 85 per cent correct, with zero cases that disclose another supplier's data." The second half matters: some failures are not to be traded against averages.
Alongside quality, state the targets for speed, cost and safety. For the invoice agent: under two minutes per query, under ₹6 per query in model and system costs, and no disclosure of another supplier's data in any test or in production.
Does it pay? The arithmetic of resolution
The business case is a short calculation. Define three things:
H, the cost of a person handling one query. At 6 minutes and ₹300 an hour this is ₹30.
A, the cost of an agent attempt, including model calls and systems. Say ₹4.
r, the resolve rate: the share of queries the agent completes correctly without a person.
If a query the agent cannot resolve is passed to a person, who then does the whole job, the average cost per query is
A + (1 − r) × H
At r = 0.85 this is 4 + 0.15 × 30 = ₹8.50 against ₹30 for the person alone. The saving per query is ₹21.50, and at 300 queries a day it is ₹6,450 a day, about ₹1.6 lakh in a month of 25 working days.
The same formula gives the break-even resolve rate. The agent pays when A + (1 − r) × H is less than H, which means r is greater than A ÷ H = 4 ÷ 30 = 13.3 per cent. An agent that resolves even one query in eight is worth running, as long as failures cost nothing worse than a handover. This is why escalation is the quiet hero of agent design: a safe way to give up turns a mediocre agent into a profitable one. If a failure instead causes harm, the calculation of Lesson 1.3 applies and the bar is much higher.
Prepare the test cases first
Before building, assemble a test set: a collection of realistic inputs, each with the correct outcome written down. It serves three purposes: it makes "works" mean something, it guides development by showing what fails, and it catches regressions when you change the instructions or the model.
A good starting set has 30 to 50 cases drawn from real history where possible, covering four kinds.
Typical cases, in the proportions they occur, perhaps two thirds of the set.
Edge cases: invoices not found, two invoices in one message, a number with a typo, a supplier asking about someone else's invoice.
Cases that should escalate, to test that the agent gives up properly.
Hostile or odd cases: a message that tries to give the agent new orders, abusive language, a request in another language.
For each, record the input, the expected outcome (for example "replies with reason E12 and the fix", or "escalates") and what must never happen ("must not reveal the other supplier's invoice").
How many cases are enough? The answer is set by the margin you can live with. If the agent truly resolves 85 per cent of cases and you test on 50, the measured rate will typically land within about 1.96 × the square root of 0.85 × 0.15 ÷ 50 = 9.9 points of the truth: anywhere from 75 to 95 per cent. A test set of 50 cannot tell an agent that is 85 per cent right from one that is 78. To tell 85 from 80 reliably takes hundreds of cases. Use the small set to find bugs, the larger set to make decisions, and be honest about the margin when reporting.
Cost, speed and risk estimates
Complete the design with three estimates that you will check against reality later.
1. Steps per query, from the task map. For the invoice job: about 4 to 6 model calls.
2. Tokens and cost per query, using the method of Lesson 1.2: the instructions and tool descriptions are read at every step, and the history grows.
3. The worst case. What is the longest plausible run, and what does it cost? Set the step and budget limits just above the normal range.
Then list the risks and assign each a control: wrong disclosure (a permission check in the tool, not the prompt), wrong refund promise (the agent has no such tool), looping (the step limit), poor tone (a review of samples), data protection (logs with personal details kept under policy).
Designing in code: a specification you can test
Writing the design as a structure makes it testable, and the same structure drives the build in Lesson 3.4. The following sketch holds the job description, a function for the cost model and a way to check a test set's margin.
python
import math
JOB = {
"purpose": "Answer suppliers' questions about invoice status and rejection reasons.",
"authority": {"may": ["read invoices", "read rejection rules", "reply to the supplier"],
"maynot": ["change an invoice", "promise a payment date", "discuss another supplier"]},
"tools": ["getinvoice", "getrule", "escalate", "finish"],
"escalatewhen": ["invoice not found", "unknown rejection code", "angry or legal message",
"request outside authority"],
"success": {"correctandcomplete": 0.85, "disclosures": 0, "maxcostrupees": 6},
}
TASKMAP = [
("find invoice number in message", "model"),
("check number format", "code"),
("look up invoice", "tool"),
("look up rejection rule", "tool"),
("explain and say what to fix", "model"),
("send reply", "tool"),
("disputed amount", "person"),
]
def costperquery(personcost, agentcost, resolverate):
"""Average cost when unresolved queries go to a person who does the whole job."""
return agentcost + (1 - resolverate) personcost
def breakeven(personcost, agentcost):
return agentcost / personcost
def margin(passrate, n, z=1.96):
return z math.sqrt(passrate (1 - passrate) / n)
print(round(costperquery(30, 4, 0.85), 2), round(breakeven(30, 4), 3))
print([round(costperquery(30, 4, r), 1) for r in (0.5, 0.7, 0.85, 0.95)])
for n in (30, 50, 200, 1000):
print(n, round(100 margin(0.85, n), 1))
the share of duties that need the model decides how agentic the job really is
sharemodel = sum(1 for , who in TASKMAP if who == "model") / len(TASKMAP)
print(f"{sharemodel:.0%} of duties need the model; the rest are code, tools or people")
Run it and edit the numbers. The model share of the task map is a useful conversation starter: if only a quarter of the duties need judgement, the design should look more like a workflow than a free agent.
Common mistakes
Building before defining the job. Without a written purpose, authority and success measure, the project drifts.
Giving the model every duty. Put exact steps in code, facts in tools, and authority with people.
Choosing maximum freedom first. Start with a fixed path and a safe handover, and add freedom as the evidence earns it.
No success measure, or one that counts activity rather than outcomes.
Test cases written after building. They end up reflecting what the agent already does.
A test set with no escalation or hostile cases. The failures that matter most are never tested.
Over-reading a small test. Fifty cases leave a margin of about 10 points. Say so.
Ignoring the handover cost. The economics depend on what happens to the queries the agent cannot resolve.
Skipping the risk list. Each risk needs a named control, ideally in code.
Key takeaways
Design an agent as you would define a job: purpose, inputs, outputs, duties, authority, tools, success measure and escalation, on one page a manager can check.
Map each duty to code, a tool, the model or a person. The map shows how agentic the job really is. Often a workflow with a handover is the right first version.
Define success as outcomes you can check, with thresholds fixed in advance and zero-tolerance failures stated separately.
Cost per query = agent cost + (1 − resolve rate) × person cost. In the example, ₹4 + 0.15 × ₹30 = ₹8.50 against ₹30, and the agent breaks even at 13.3 per cent.
Prepare the test set first: typical, edge, escalation and hostile cases. A set of 50 leaves a margin of about 10 points around 85 per cent.
Estimate steps, tokens, cost and the worst case, and give each risk a control.
Check yourself
1. Name three parts of a job description for an agent.
For example purpose, authority and success measure. The others are inputs, outputs, duties, tools and escalation.
2. A person handles a query for ₹40. An agent attempt costs ₹5 and resolves 80 per cent, passing the rest to a person. What is the average cost per query?
5 + 0.2 × 40 = ₹13.
3. What resolve rate does the agent in question 2 need to break even?
5 ÷ 40 = 12.5 per cent.
4. An agent passes 36 of 40 test cases. What is the measured rate, and the margin of error?
90 per cent. The margin is 1.96 × the square root of 0.9 × 0.1 ÷ 40 = 9.3 points, so the true rate could be about 81 to 99 per cent.
5. In a task map, five of eight duties are exact lookups and formatting. What does this suggest?
The job is mostly a workflow. Use code and tools for those steps and use the model only where judgement or language is needed.
Try this
Set aside ninety minutes. You will build on the result in the next three lessons, so choose a job you care about.
1. Pick a job for your first agent that is small, frequent and checkable. Write the eight-part job description.
2. List its duties and build the task map. Count how many need the model.
3. Decide whether a fixed workflow with a handover would do for the first version. Write down the one or two places where you would want the agent to choose.
4. Estimate H, A and r for your job. Use the economics simulator to find the average cost, the saving a day and the break-even resolve rate.
5. Write 30 test cases: typical, edge, escalation and hostile. For each, note the expected outcome and anything that must never happen.
6. Use the test-set simulator to find the margin for your set, and decide how many more cases you would need to compare two versions that differ by 5 points.
7. List five risks and a control for each, saying whether the control is in a tool, in the harness or with a person.
A good result is a one-page job description, a task map, a cost estimate with a break-even point, a set of test cases written before any code and a risk table with controls. The next lesson turns the description into instructions the model can follow.
Tags: Agentic AI Agent design Job description Agent evaluation AI agents