Lesson 1.4: The parts of an agent: model, tools, memory and instructions
Learn the five parts every agent is built from, what each does, how they share a limited context window, and how to write the plan for an agent on one page.
By the end of this lesson you will be able to name the five parts of an agent, say what each part is responsible for, describe how the parts share a limited context window, budget that window with simple arithmetic, and draft a one-page specification for an agent you could hand to an engineer. You will work through a support agent whose fixed parts take 6,500 tokens, whose first twenty steps add 20,000 more, and which leaves 73,500 tokens free in a 100,000-token window, and you will see how twelve well-described tools cost 3,000 tokens while forty cost 10,000. You should have read Lessons 1.1 to 1.3, which define agents, the loop and the levels of autonomy. The mathematics is addition and multiplication.
You now know what an agent is, how its loop runs and how much freedom to give it. This lesson opens the box and looks at what an agent is made of. There are five parts: the model, the instructions, the tools, the memory and the harness. Every agent has all five, even when a framework hides some of them. Knowing them by name lets you describe any agent in a few sentences, find what is wrong when one misbehaves, and design a new one without starting from a blank page.
The five parts
1. The model
The model is the language model that reads text and produces text. In an agent it plays the role of decision-maker: given everything it can see, it chooses the next action. Three properties of a model matter most for agent work.
Instruction following. Can it do exactly what a long, detailed instruction says, including what not to do?
Tool use. Can it produce well-formed tool requests with the right arguments, and read the results sensibly?
Reasoning over several steps. Can it keep a goal in mind, notice that a step failed and change course?
Models also differ in speed, in cost per token and in how much text they can read at once. These differences are large, and the right model is often not the biggest one. A common design uses a more capable model for planning or hard judgements and a smaller, cheaper, faster one for routine steps such as extracting a date from a document. Module 8 covers choosing a model for each step. In this lesson, remember that the model is only one of five parts. A better model helps, but a poor description of the tools or a missing limit in the harness will undo it.
2. The instructions
The instructions, often called the system prompt, tell the model who it is, what its goal is, how to behave and what the rules are. They are the same for every run, and they sit at the top of the text the model reads at every step. A good set of instructions for an agent usually covers:
Role and goal: "You help the accounts team find out why invoices are rejected."
Process guidance: "First look up the invoice, then look up the rejection code, then compare them. If the invoice cannot be found, say so."
Boundaries: "Never change a record. Never share one customer's data in an answer about another."
Output format: "Finish with a two-sentence summary and the code you found."
What to do when stuck: "If you are unsure, ask for help by calling escalate with a short reason."
Instructions are the cheapest and fastest part to change, which makes them tempting to overuse. Adding rules to the instructions for every new failure produces long, tangled prompts that the model follows unevenly. A better habit is to fix a problem where it lives: improve a tool description, add a check in the harness, or split the task, and keep the instructions short enough to read in a minute. Lesson 3.2 teaches the craft of writing them.
3. The tools
Tools are the ways the agent can find things out and do things. Each tool has three visible parts, the ones the model reads:
a name, such as getinvoice;
a description in plain words, such as "Returns the status, amount and rejection code for one invoice. Use when you have an invoice number.";
a list of arguments with their types, such as invoicenumber, a string of digits.
Behind those three parts is ordinary code that the harness runs, and the model never sees it. The description is what the model uses to decide whether and how to call the tool, which makes it one of the most important texts in the whole system. Tools fall into three groups: ones that read (search, look up, fetch a page), ones that write or act (send, update, pay) and ones that compute (run code, calculate). Reading tools are usually safe to give freely. Writing tools need care. Module 2 is devoted to this topic.
4. The memory
An agent has several kinds of memory, and it helps to separate them.
Kind · What it is · How long it lasts · Where it livesWorking memory · The text the model reads in this run: instructions, history, tool results · The current run · The context window
Run state · Facts the harness tracks, such as steps used, money spent, which sub-tasks are done · The current run · Variables in the harness
Long-term memory · Facts saved for later runs, such as a customer's preferences or a note on past mistakes · Across runs · A file or database that the agent can search
Knowledge · Documents and data the agent can look up · Permanent · An external store, often searched by meaning
The first row is the most important to understand and the most often misunderstood, because the model itself remembers nothing from one call to the next. Whatever it needs to know must be in the context window at the moment it decides, or the agent must be able to fetch it. Module 4 goes through how to manage each kind.
5. The harness
The harness is the program that runs everything else: it builds the prompt, calls the model, parses its reply, runs tools, appends results, checks the stop rules and enforces permissions. It is also called the runtime, the scaffold, the orchestration layer or the control loop. Frameworks and software development kits provide a harness so that you do not write one from scratch, though you can.
The harness is where the engineering that makes an agent trustworthy lives. It enforces the step limit and the budget. It validates that a tool request is well formed before running it. It asks a person for approval before an action above the permitted level of autonomy. It writes the trace of every step so that someone can later see what happened. It catches errors, such as a tool that fails or hangs, and turns them into messages the model can act on. A strong harness around an average model is usually safer than a clever model in a weak harness.
Predict first: a team says "our agent keeps making up invoice numbers." Which of the five parts would you check first: the model, the instructions, the tools, the memory or the harness? Decide before reading on.
Check the tools first, and then the harness. Invented identifiers usually mean that the model had no reliable way to find the real ones. Perhaps the search tool's description does not say it returns invoice numbers, or the result does not include them, so the model fills the gap. A harness check that rejects any invoice number the tools have not returned would also stop it. Switching to a bigger model might reduce the problem without removing its cause. The general habit is to look for the cause in the surroundings of the model before blaming the model itself.
How the parts fit together
The parts meet in one place, the context window: the block of text the model reads when it decides. At every step the harness fills it with the instructions, the descriptions of all the tools, relevant memory, the goal and the history of the run so far. Whatever is not in the window does not exist for the model.
The window is limited. To make the following calculation concrete, suppose a model can read 100,000 tokens at once. (Real limits vary widely between models and change over time, so use the figure from your own model's documentation.) A token is a piece of text, on average about three quarters of an English word, or four characters.
Consider a support agent. Its fixed parts, the ones present at every step, are:
the instructions, 1,500 tokens (roughly 1,100 words);
twelve tools, each with a name, a description and arguments, about 250 tokens each, which is 3,000 tokens;
a block of long-term memory retrieved for this customer, 2,000 tokens.
These total 6,500 tokens before the run has done anything. Now the history grows. Suppose that after twenty steps the goal, the model's notes and the tool results have added 20,000 tokens. The window now holds 26,500 tokens, which leaves 73,500 for further history and the model's reply.
Three design lessons come from the budget.
Tools are not free. Every tool you give the agent costs tokens at every step, whether or not it is used. Twelve tools at 250 tokens is 3,000. Forty tools would be 10,000, a tenth of the window, and they carry a second cost that does not show in the token count: more tools mean more choices, and a model with forty tools to pick from chooses wrongly more often than one with twelve. Give an agent only the tools its job needs, and consider grouping them by task so that each sub-agent sees only its own.
History is the part that grows. Everything else is fixed per run. Left alone, the history will eventually fill the window, so the harness must trim, summarise or move work to a sub-agent before that happens. The previous lesson showed that the cost of reading also rises with each step.
Long contexts are not equally well used. Even when the window has room, models tend to pay less attention to material in the middle of a very long context. A window that is 90 per cent full is a warning sign even if nothing has yet been cut off.
Mapping an agent onto the parts
A one-page specification forces you to decide each part before building. Here is one for the support agent.
Part · DecisionGoal · Resolve fee disputes on personal loans without a person for routine cases
Model · A capable model for decisions, a small one for extracting fields from messages
Instructions · Role, a four-step process, the refund limit, tone rules, an escalation rule. About 1,500 tokens
Tools · getloan, getpayments, getfeepolicy, reversefee (limit ₹500), sendreply, escalate. Six in all
Memory · Working: this conversation. Long-term: the customer's past disputes. Knowledge: the fee policy documents
Harness · Step limit 12, budget limit, approval needed above ₹500, trace stored, tool timeout of 10 seconds
Autonomy · Reading: level 4. Replying and refunds up to ₹500: level 3. Anything above: level 2
Success measure · 90 per cent of routine disputes closed correctly, checked by sampling
Failure plan · On any doubt, escalate to a person with the trace attached
This table is short, and it is worth more than many pages of code written without it. It states the level of autonomy from Lesson 1.3, anticipates the cost from Lesson 1.2 and lists the tools whose design Module 2 teaches. Engineers can build from it, and reviewers can challenge it.
The parts in code
The same specification can live in code as a plain structure. This is useful because it can be reviewed, versioned and tested like any other configuration. The helper below also estimates the window budget using the rough rule of four characters per token.
python
AGENT = {
"goal": "Resolve routine late-fee disputes on personal loans.",
"model": {"decide": "capable-model", "extract": "small-model"},
"instructions": (
"You are a support agent for a lender. Look up the loan and the payments, "
"compare the payment date with the due date, and reverse the fee only if the "
"payment arrived on or before the due date. Never reverse more than 500 rupees. "
"If anything is unclear, call escalate with a short reason."
),
"tools": {
"getloan": "Returns status, due date and fee for one loan. Use when you have a loan number.",
"getpayments": "Returns payments for one loan with dates and amounts.",
"reversefee": "Reverses one fee. Allowed only up to 500 rupees. Returns confirmation.",
"escalate": "Hands the case to a person. Arguments: a reason of one sentence.",
},
"harness": {"maxsteps": 12, "maxcostrupees": 15, "approvalaboverupees": 500, "tooltimeouts": 10},
"autonomy": {"read": 4, "reply": 3, "refundsmall": 3, "refundlarge": 2},
}
def tokens(text):
return len(text) // 4 # rough: four characters per token
def windowbudget(agent, window=100000, memory=2000, history=20000):
fixed = tokens(agent["instructions"])
fixed += sum(tokens(name + desc) + 40 for name, desc in agent["tools"].items()) # 40 for the argument list
fixed += memory
used = fixed + history
return {"fixed": fixed, "used": used, "free": window - used, "shareused": round(100 used / window, 1)}
print(windowbudget(AGENT))
guard: every tool the model can see must have a plain description and a limit in the harness
for name, desc in AGENT["tools"].items():
assert len(desc) > 20, f"tool {name} needs a real description"
assert AGENT["harness"]["maxsteps"] <= 20, "step limit too generous for this task"
Reading the specification back as code makes gaps visible. A tool with a thin description, a missing limit or an autonomy level that does not match the tool's harm all show up when the structure is written down.
Common mistakes
Treating the model as the whole agent. Four other parts decide whether it works. Check tools and the harness before swapping models.
Long, tangled instructions. Fix problems where they live, in tools and checks, and keep instructions short enough to read in a minute.
Vague tool descriptions. The description is what the model uses to choose. Say what the tool does, when to use it and what it returns.
Too many tools. Each costs tokens at every step and adds ways to choose wrongly. Give only what the job needs.
Assuming the model remembers. Anything it must know has to be in the window or fetched by a tool.
A weak harness. No step limit, no timeouts, no validation and no trace leave problems invisible and unstoppable.
Ignoring the window budget. History grows. Plan trimming before the window fills.
No written specification. Without one page, parts get decided by accident.
Key takeaways
Every agent has five parts: the model decides, the instructions set role and rules, the tools give it abilities, the memory carries what it needs to know, and the harness runs the loop and enforces limits.
The model remembers nothing between calls. Everything it uses must be in the context window at the moment it decides.
Tools are described by a name, a description and arguments, and the description is what the model reads. Reading tools are safe to give freely and writing tools need care.
The context window is a shared budget. In the example, instructions (1,500), twelve tools (3,000) and memory (2,000) use 6,500 tokens before any history, and twenty steps add 20,000, leaving 73,500 of 100,000.
More tools cost tokens at every step and increase wrong choices. Forty tools at 250 tokens each would take 10,000.
A one-page specification, covering goal, model, instructions, tools, memory, harness, autonomy, success measure and failure plan, is the best first design tool.
Check yourself
1. Which of the five parts enforces a step limit?
The harness. It runs outside the model, so it can stop the loop even when the model does not want to stop.
2. A model has a window of 50,000 tokens. Instructions take 1,000, eight tools take 200 each, memory takes 1,400 and the history is 12,000. How many tokens are free?
Fixed: 1,000 + 1,600 + 1,400 = 4,000. Used: 16,000. Free: 34,000.
3. Why can giving an agent forty tools make it worse than giving it twelve?
They use window space at every step, and the model has more choices, so it picks the wrong tool more often.
4. A customer's preferences are stored from last month and loaded into the next conversation. Which kind of memory is this?
Long-term memory. It is saved outside the run and retrieved into the window when needed.
5. Name two things a good tool description should contain.
What the tool does, when to use it, and what it returns. Also the meaning of its arguments.
Try this
Set aside sixty minutes. This is the first design exercise of the course, and you will reuse it in later modules.
1. Choose a job you would like an agent to do. Write its goal in one sentence.
2. Fill in the specification table from this lesson, one row for each part. Keep every cell short.
3. List the tools. For each, write the name, a one-sentence description with "use when" and "returns", and its arguments. Mark each as read, write or compute.
4. Draft the instructions in under 250 words. Mark any rule that would be better enforced by the harness.
5. Estimate the window budget with the context simulator, using your number of tools and a guess for history after 15 steps.
6. Set an autonomy level for each action using Lesson 1.3, and write the stop rules from Lesson 1.2.
7. Write the failure plan: what the agent should do when it is stuck, and who receives the case.
A good result is a one-page specification that an engineer could build from, with every part decided, a window budget and an autonomy level for each action. The next module turns to the tools themselves, because that is where most agents succeed or fail.
Tags: Agentic AI Agent architecture Context window Tools Agent memory