Lesson 4.3: Managing context: short-term memory and summaries
Learn to keep an agent's context useful as a run grows: trimming, sliding windows, rolling summaries, pinned notes and offloading, with the token arithmetic.
By the end of this lesson you will be able to explain why a long agent run needs active management of what the model reads, compare the main techniques for doing it, work out how many tokens each saves, write a good summary note that preserves what matters, and recognise the ways context management itself can go wrong. You will take a 40-step run in which every step adds 800 tokens and show that the model reads 704,000 tokens in total if nothing is managed, 356,000 with a sliding window of ten steps and 272,000 with a rolling summary, while the amount read at the last step falls from 33,200 tokens to 10,000 or 11,600. You should have read Lessons 1.2 and 1.4, which explain how the history grows and how the parts share the context window. A little Python helps for the code example. The mathematics is repeated addition.
In Module 3 your agent ran for two or three steps, so its context was never a concern. Real work is longer: an investigation across several systems, a research task over dozens of sources, a code change that touches many files. As the run lengthens, the history that the harness resends at every step grows with it, and three problems appear together. The cost rises faster than the number of steps. The window eventually fills and the run fails. And well before it fills, the model starts to do worse, because the important facts are buried among thousands of tokens of old tool output. Context management is the set of techniques for keeping the right information in front of the model and the rest out of its way. It is called short-term memory management, because it deals with what the agent remembers within one run.
The problem in numbers
Return to the arithmetic of Lesson 1.2. Suppose the instructions and tool descriptions take 2,000 tokens, and every step adds 800 tokens: a tool call, its result and the model's note. At step k the model reads 2,000 + 800 × (k − 1) tokens. At step 1 that is 2,000. At step 10 it is 9,200. At step 20 it is 17,200, and at step 40 it is 33,200. The total over 40 steps is 40 × 2,000 + 800 × (40 × 39 ÷ 2) = 80,000 + 624,000 = 704,000 tokens.
That is the cost of doing nothing. Notice two things. The total grows with the square of the number of steps, so doubling the length of a run more than triples what the model reads. And the last steps are the most expensive, because they carry everything that came before.
Predict first: a run has 40 steps. Keeping only the last ten steps in the context (a sliding window) means the model forgets everything older. Do you expect this to cut the total tokens read by about a tenth, about a half, or by about nine tenths? Decide before reading on.
About a half. With a window of ten steps, the first ten steps read the same as before, and from then on every step reads a fixed 2,000 + 800 × 10 = 10,000 tokens. The total is 356,000, which is 51 per cent of 704,000. The saving is large, nearly half, but not nine tenths, because the early steps are unchanged and every later step still carries ten steps of history. The saving grows with the length of the run: at 100 steps the window reads about 23 per cent of what an unmanaged context would.
The techniques
1. Keep results small at the source
The cheapest fix is the one from Lesson 2.2: return less in the first place. If the typical step adds 800 tokens because a tool returns 600 tokens of result, trimming results to 250 tokens cuts the addition to 450. This reduces every strategy below and, unlike them, loses nothing, because the agent never sees the extra text. Do this first.
2. Sliding window
Keep the instructions, the goal and the last w steps, and drop the rest. It is simple and predictable: the context never exceeds a fixed size. Its weakness is plain. Anything older than the window is forgotten, including facts the agent may need: the supplier's name found at step 2, a decision made at step 5, a constraint from the original request. A window is only safe when old steps rarely matter, or when it is combined with something that keeps the important facts.
3. Rolling summary (compaction)
Instead of dropping old steps, summarise them. When the history reaches a threshold, the harness asks a model call to condense the older steps into a short note, and replaces them with it. The context then holds the instructions, the summary and the recent steps. Suppose the agent summarises every ten steps, compressing eight thousand tokens of new history into eight hundred (a ratio of ten to one). Then the summary grows by 800 tokens per ten steps, and the recent steps add at most 8,000. The model reads about 9,200 tokens at step 10, 10,000 at step 20, 10,800 at step 30 and 11,600 at step 40. The total is 272,000 tokens, 39 per cent of the unmanaged 704,000 and below the sliding window's 356,000, and the early facts are not lost, only condensed. The price is one extra model call per summary, and the risk that the summary leaves something out.
4. Pinned notes (a scratchpad)
A different approach is to give the agent a place to write down what matters as it goes: a notes block that the harness keeps in the context at every step, outside the trimmed history. The agent adds short entries: "Supplier is S1, invoice 8841, rejected E12, rule says missing purchase order." Whatever else is trimmed or summarised, the notes remain. This works well because the model itself decides what is worth keeping, at the moment it learns it. Two disciplines make it work: keep the notes short and structured, and have the harness cap their size so that the notes do not become a second history.
5. Offload to storage and fetch on demand
For bulky material, such as a long document or a large table, do not put it in the context at all. Save it where the agent can reach it, in a file or a store, and put in the context only a short reference and a description: "Saved the 400-row payment list as paymentsoct.csv; columns are date, amount, reference." Give the agent tools to read parts of it or to compute on it. The context holds the pointer, and the data stays outside. This combines naturally with the code-execution tool of Lesson 2.4: the agent filters the file with a few lines of code and reads only the answer.
6. Fresh start with a hand-off note
When a run has become long and cluttered, it can be better to stop and restart with a clean context that contains only the instructions, the goal and a concise hand-off note summarising what has been done and what remains. This is rolling summary taken to its limit. It is also how long tasks survive a crash or a pause: the note is the saved state.
7. Sub-agents with their own context
A task can be split so that a sub-agent does a self-contained piece of work in its own clean context and returns only a short result. The main agent's context never sees the sub-agent's many steps, only its answer. This is one of the strongest reasons to use sub-agents, and Module 5 covers it.
Technique · Saves · Loses · Extra cost · Best forTrim at source · Tokens on every step · Nothing the agent needs, if done well · None · Always
Sliding window · A lot; fixed size · Everything older than the window · None · Runs where old steps rarely matter
Rolling summary · A lot; slowly growing size · Detail not captured in the summary · One call per summary · Long runs where early facts matter
Pinned notes · Keeps key facts, not history · Whatever the agent forgets to note · A few tokens per note · Any run with facts to carry forward
Offload to storage · Bulk data · Nothing, if fetch tools are good · Fetch calls · Large documents and tables
Fresh start with note · Everything but the note · Anything the note omits · One summary call · Very long runs, pauses, recovery
Sub-agent · The sub-task's steps · Detail beyond its returned answer · Extra calls · Self-contained sub-tasks
## Writing a good summary
A summary is a lossy copy, and what it loses is chosen by the prompt that makes it. A careless instruction such as "summarise the conversation so far" produces a pleasant paragraph that omits the exact figures, identifiers and decisions that the agent needs. A good summary is structured and states what to preserve:
Section · What to keepGoal · The original request and any constraints, in the user's words
Done · Steps completed, each with its result in a line
Facts · Exact identifiers, names, figures, dates and their sources, copied and not paraphrased
Decisions · Choices made and the reasons, so they are not reopened
Open items · What is still unknown or in progress
Next step · The very next action, so the agent can resume at once
Here is the difference in practice. After fifteen steps of investigating a rejected invoice, a careless summary might read: "The agent looked into the supplier's invoice problem, found a few issues with the purchase order and is close to an answer." Nothing in it can be acted on. A structured note reads:
Goal: Explain to supplier S1 why invoice 8841 was rejected. Constraint: do not contact the supplier.
Done: Invoice looked up (rejected, code E12, amount 48,000). Rule E12 looked up.
Facts: Rule E12 text: "Purchase order number missing or invalid." Invoice has no purchase order field.
Decisions: Not checking payments, because the invoice was never approved for payment.
Open items: Whether the supplier's other invoice, 8842, has the same problem.
Next step: Look up invoice 8842, then write the reply.
An agent that wakes up with this note can carry on at once. An agent that wakes up with the vague paragraph has to start again.
The central rule is: copy exact values; summarise narrative. "Invoice 8841, amount 48,000, rejected E12" survives as it is. A paragraph of the supplier's story can be shortened to a line. A number that is paraphrased ("about forty thousand") may be wrong when used later.
Summaries have their own failure modes.
Drift. Each summary of a summary loses a little, and details degrade. Keep the exact facts in pinned notes so that they are never re-summarised.
Baked-in errors. If the summary states something wrong, every later step trusts it. Include sources, and verify critical facts against the original.
Lost instructions. A summary that drops a constraint such as "do not contact the supplier" removes a safety rule. Pin the constraints with the goal.
Over-compression. Ten-to-one may be too aggressive for fact-dense material. Test with questions.
Recency bias. Summarisers favour what happened last. Ask explicitly for balance.
Always keep the full original trace in storage, even when the context holds only a summary. If a later step needs a detail that the summary dropped, a search tool can retrieve it from the trace. This turns the loss from permanent to recoverable.
Where to put things
The position of text in a long context matters. Models tend to attend best to the start and the end, and less to the middle. So the harness should build the context in a deliberate order: instructions and constraints first, then the goal, then the summary and pinned notes, then the recent steps, and the current question last. Keep the layout stable from step to step. A stable, append-only prefix also lets providers that offer cheaper pricing for repeated text reuse it, while rewriting earlier text on every step defeats that saving, so compact at intervals instead of continuously.
A context manager in code
python
class Context:
"""Keeps instructions, pinned notes, a rolling summary and the most recent steps within a token budget."""
def init(self, instructions, budget=6000, keeplast=4):
self.instructions, self.budget, self.keeplast = instructions, budget, keeplast
self.notes, self.summary, self.recent = [], "", []
@staticmethod
def tokens(text):
return len(text) // 4 # rough rule: four characters per token
def note(self, text):
self.notes.append(text) # pinned: never summarised away
def add(self, text):
self.recent.append(text)
if self.size() > self.budget:
self.compact()
def summarise(self, entries):
"""Stand-in for a model call: keep exact FACT lines, shorten everything else."""
facts = [line for e in entries for line in e.splitlines() if line.startswith("FACT:")]
gist = [e.splitlines()[0][:60] for e in entries]
return "Earlier: " + "; ".join(gist) + "\n" + "\n".join(facts)
def compact(self):
old, self.recent = self.recent[:-self.keeplast], self.recent[-self.keeplast:]
self.summary = (self.summary + "\n" + self.summarise(old)).strip()
def render(self):
return "\n\n".join(part for part in (
self.instructions, "NOTES\n" + "\n".join(self.notes) if self.notes else "",
"SUMMARY\n" + self.summary if self.summary else "", "RECENT\n" + "\n".join(self.recent)) if part)
def size(self):
return self.tokens(self.render())
ctx = Context("You are a careful accounts agent. " 20, budget=4000)
ctx.note("Goal: explain why invoice 8841 was rejected. Constraint: do not contact the supplier.")
unmanaged = 0
for step in range(1, 41):
body = f"Step {step}: tool result\nFACT: item {step} amount {1000 + step}\n" + "x" 2500 # about 640 tokens of bulk
unmanaged += Context.tokens(ctx.instructions) + 20 + step 640 # what a plain history would send
ctx.add(body)
print("final context tokens:", ctx.size(), "budget:", ctx.budget)
print("facts kept:", sum(f"FACT: item {n} " in ctx.render() for n in range(1, 41)), "of 40")
print("constraint still present:", "do not contact the supplier" in ctx.render())
print("approximate tokens an unmanaged history would read at step 40:", Context.tokens(ctx.instructions) + 40 640)
Run it. The context stays near its budget however long the run goes, all forty exact facts survive in the summary or the recent steps, and the pinned constraint is never lost. The stand-in summariser shows the principle of copying exact values and shortening narrative. A real summariser would be a model call with a prompt built on the table above.
Common mistakes
Doing nothing until the window overflows. Cost and quality suffer long before failure. Plan management from the start.
Large tool results. Trim at the source before any other technique.
A window with no memory of old facts. Pair it with pinned notes or a summary.
Careless summaries. Name what to preserve, and copy exact values.
Summaries of summaries. Keep exact facts in pinned notes, outside the summarising chain.
Losing constraints. Pin the goal and the rules so they survive every compaction.
Discarding the original trace. Keep it in storage so details can be retrieved.
Compacting on every step. It adds cost and defeats caching. Compact at thresholds.
Unbounded notes. Cap the notes block, or it becomes a second history.
Key takeaways
A long run's context grows with every step, and the total read grows with the square of the number of steps. Unmanaged, a 40-step run at 800 tokens a step reads 704,000 tokens, and the last step alone reads 33,200.
First trim results at the source. Then choose among a sliding window, a rolling summary, pinned notes, offloading to storage, a fresh start with a hand-off note, and sub-agents.
A window of ten steps cuts the total to 356,000. A rolling summary every ten steps cuts it to 272,000 while keeping early facts in condensed form.
Pinned notes keep the facts the agent chose to record, outside the trimmed history.
A good summary is structured: goal, done, facts, decisions, open items, next step. Copy exact values and shorten narrative.
Summaries drift and can bake in errors or drop constraints. Pin constraints, keep the original trace and verify critical facts.
Put instructions first, key notes next and the current question last, and compact at thresholds, not continuously.
Check yourself
1. A run has 2,000 fixed tokens and adds 500 per step. How many tokens does the model read at step 21 with no management?
2,000 + 500 × 20 = 12,000 tokens.
2. Using a window of 5 steps, what is the most the model reads at any step in question 1's run?
2,000 + 500 × 5 = 4,500 tokens.
3. Why should exact identifiers and figures be copied into a summary, not paraphrased?
A paraphrase such as "about forty thousand" may be wrong when used later. Exact values must survive unchanged.
4. Name two ways that summaries can go wrong.
For example drift as summaries of summaries lose detail, baked-in errors that later steps trust, or dropped constraints.
5. A task needs a 400-row table. How should it be handled in the context?
Save it to a file or store, and keep only a short reference and description in the context, with a tool to read or compute on parts of it.
Try this
Set aside seventy-five minutes.
1. Take your agent and estimate the fixed tokens and the tokens added per step from a real or imagined trace. Find the step at which the window would overflow with no management.
2. Use the simulator to compare the total reading for your run with windows of 5, 10 and 20 steps.
3. Trim your largest tool result to the fields the task needs. Estimate the new tokens per step.
4. Write the summary prompt for your agent using the six sections in this lesson. Decide which values must be copied exactly.
5. Decide what belongs in the pinned notes, including the constraints, and set a cap on their size.
6. Run the context-manager code. Change the budget and the keeplast value, and see what happens to the facts kept. Break it by removing the pinned constraint and note what is lost.
7. Write the rule for when to compact, when to restart with a hand-off note, and what you store permanently.
A good result is a context plan with a token estimate for each strategy, a structured summary prompt, a list of pinned facts and constraints, and a stored trace. The next lesson covers memory that lasts beyond one run: how an agent remembers across sessions.
Tags: Agentic AI Context management Agent memory Summarisation Context window