Lesson 8.2: Caching and batching to serve more requests
Batching lets a model serve many requests at once, and caching avoids repeat work. Watch static, dynamic and continuous batching schedule the same six requests.
When a product gets busy, two ideas decide how many users one set of hardware can serve: doing work for several requests together, and not doing the same work twice. They are called batching and caching. By the end of this lesson you will be able to explain why a model serves several requests at once, tell static, dynamic and continuous batching apart and say which one keeps users waiting least, work out how much an answer cache or a prompt cache could save in rupees, and spot the rules that keep a cache from serving a wrong or private answer. You need no engineering background beyond the earlier lessons in this track. The numbers in the simulators are illustrative, chosen to show the shape of each trade-off, not to quote any real system or price.
Why batch at all
A graphics processor can do a very large amount of arithmetic at once. Running one request at a time leaves most of that capacity idle. Handling several requests together lets the machine share its effort, so more users can be served in the same time. The question is how to group them, and that is where the strategies differ.
An everyday comparison helps. Imagine a tea stall with one large kettle. Boiling it for a single cup wastes most of the heat and time, since the kettle takes about as long to boil for one cup as for six. A sensible stall owner waits a moment, collects a few orders and boils for all of them. But there is a limit: if the owner waits too long for a full kettle, the first customer is left standing, and may walk off. Batching is the same bargain. More requests per run means better use of the machine, and waiting to collect them means a delay for everyone who arrived early.
This bargain, throughput against waiting time, is the heart of the lesson. Everything else is about getting the best of both.
Three ways to batch
Static batching waits until a fixed number of requests have arrived, runs them together, and returns every answer only when the slowest one finishes.
Dynamic batching launches when the batch is full or after a short wait, whichever comes first. It stops early requests waiting forever, but answers are still held until the slowest in the batch is done.
Continuous batching lets each request finish on its own. The moment one leaves, the next waiting request takes its place. Nobody waits for a slower neighbour.
The difference matters most for language models, because requests are not equal in length. One user asks for a one-line answer and another asks for a long essay. In static and dynamic batching the short request has finished its work but must sit until the long one is done. Continuous batching removes that wasted wait. Here is the comparison in a table.
Strategy · When a run starts · Who waits · Typical weaknessStatic · When a full batch has arrived · Early arrivals, then short requests held for long ones · Idle time waiting for a full batch, and held-back answers
Dynamic · When the batch is full or a timer runs out · Short requests held for long ones · Smaller batches when traffic is thin
Continuous · As soon as there is a free slot · Mostly only what the work itself needs · More complex to build and run
The simulator below runs the same six requests under each strategy. Press Play, switch tabs, and compare the bars. Grey is waiting, colour is processing, and the dashed outline is an answer that is finished but held back.
Predict first: which strategy will have the lowest average wait, and which request suffers most under static batching? Decide, then check the table under the chart.
Continuous batching wins by a wide margin: an average wait of 7.8 units, against 19.7 for static and 18.0 for dynamic. Under the static strategy the first request has to sit idle until the third arrives, then waits again for the 14-unit request in its batch.
Reading the numbers
Notice that dynamic batching improves the average wait slightly, but finishes the last request a unit later than static in this example (31 against 30), because it launches smaller batches. It helps early requests start sooner, yet it does not solve the problem of short requests being held for long ones. Continuous batching finishes everything by 19 and keeps the machine busy. The simulator is idealised: it assumes a batch runs at the same speed whatever its size, which real systems do not, so treat the gaps as a picture of the idea rather than a forecast.
Why does that last assumption matter? Because in a real system, adding requests to a batch does make each step take a little longer. The machine does more work per step, even though it does it far more efficiently than running the requests one at a time. The next simulator puts a number on that, using a figure you set yourself.
The throughput and latency trade-off
Larger batches serve more requests per second but make each step slower. Whether that is good depends on whether you care more about total work done or about how quickly each person gets a reply. The simulator below models a simple version. "Extra time per added request" is how much longer a step takes for every request you add to the batch, and it is the number you would measure on your own system.
With the starting numbers, a batch of 16 serves about 9.1 times as many requests per unit of time as a batch of one, while each step takes 1.75 times as long. Push the batch to 64 and the gain is 15.4 times but each step takes 4.2 times as long. The gain flattens while the delay keeps growing. That curve is why operators do not simply make batches as large as possible, and why continuous batching, which keeps the machine full without forcing everyone to wait for a full batch, is popular.
Caching: not doing the work twice
Batching makes each piece of work cheaper to run. Caching avoids doing some of it at all. It saves repeat work in two common ways.
Answer caching. If the same question is asked again and the answer does not go stale, return the saved answer without calling the model. This suits questions with fixed answers, such as opening hours.
Prompt caching. When many requests start with the same long block of text, such as a shared instruction or document, some providers can reuse the work done on that start, which cuts both delay and cost. Put the unchanging text first and the changing text last.
Be careful with answer caching: a saved answer is wrong the moment the facts change, so give every cached item an expiry time.
How much can an answer cache save?
The saving depends on one thing above all: how often the same question comes up. A customer-support assistant for a shop might get "what are your opening hours?" hundreds of times a day, while a legal-drafting tool might never see the same request twice. The share of requests answered from the cache is called the hit rate.
With 20,000 requests a day, 30 per cent answered from the cache and ₹0.50 a call, the cache avoids 6,000 calls a day and saves about ₹90,000 a month. Slide the hit rate to 10 per cent and the saving falls to ₹30,000. The saving scales directly with how repetitive the questions are, which is why the first step is to look at your real traffic and count how many requests are repeats.
How prompt caching differs
Prompt caching works at a different level. Many applications send the same long block with every request: a detailed instruction, a policy document, a set of examples. The model must read that block each time, and you are charged for reading it. If a provider can remember the work already done on the start of the prompt, later requests with the same start can reuse it, and the repeated part becomes cheaper and faster. The rule that makes this work is simple. The reusable part must come first and be identical every time, and the part that changes, such as the user's question, must come last. Put a changing date or name at the top, and the match breaks.
With a 4,000-token shared start, a 300-token question, 20,000 requests a day and half-price cached text, the input bill drops by roughly 42 per cent, from about ₹1.03 lakh to about ₹60,000 a month, a saving of about ₹43,000 on the example prices. If the shared part is small compared with the question, the saving all but disappears. That is the signal to look at prompt layout: long, stable instructions first, short changing questions last.
Keeping a cache honest
A cache that returns the wrong answer is worse than no cache, because the answer looks authoritative. Three rules keep caches safe.
Give every item an expiry. Opening hours may be valid for a week, prices for a day, a stock level for a minute. When the time is up, the cache forgets the answer and asks the model again.
Clear related entries when facts change. If a price changes, remove cached answers that mention it, instead of waiting for them to expire.
Never share private answers. An answer that includes one customer's account details must never be cached for anyone else. Cache keys should include whatever makes an answer specific, such as the user or the document, or the answer should not be cached at all.
It also helps to measure. Track the hit rate and, where you can, check a sample of cached answers against fresh ones. A hit rate that suddenly climbs may mean a bug in how keys are built, and one that sinks to zero may mean something upstream changed.
Which one should you try first?
With three ideas on the table, a short guide helps you decide where to start. The right first step depends on what your traffic looks like and who is waiting.
Your situation · Start with · WhyMany users ask the same few questions, and answers rarely change · Answer caching with an expiry · Repeated questions cost nothing after the first answer
Every request carries the same long instructions or document · Prompt caching, with the shared text first · You stop paying full price to re-read the same text
You run your own model for many users at once · Continuous batching · It keeps the machine busy without making short requests wait
Your work is not urgent, such as nightly labelling · Bulk or offline batching · Trading time for lower cost suits jobs nobody is waiting on
Traffic is low and each user is waiting live · Keep it simple, and measure first · At low volume the savings are small and the complexity is real
Notice that the last row matters. Caching and batching are optimisations, and optimisations are worth their trouble only when the numbers are big enough. A tool used by ten staff a day may not need any of them, and adding a cache would create a new thing to maintain and a new way to be wrong. A service handling hundreds of thousands of requests a day may find that one of these ideas is the difference between a viable cost and an unaffordable one.
A good habit is to measure before you build. Count how many requests repeat. Look at how much of each request is the same every time. Check how long users wait, and how much of the machine's capacity is idle. Those three measurements will point to the idea most likely to help, and will tell you afterwards whether it did.
Where this happens
You may not do all of this yourself. If you use a hosted model through an API, the provider usually handles batching on its servers, and your choices are about caching and about how you structure requests. If you run an open model on your own machines, batching is your job, and serving software designed for language models often includes continuous batching, so you may only need to configure it. In both cases the principle is the same: make good use of the machine, and do not repeat work you do not have to.
Some providers also offer a way to submit non-urgent work, such as overnight jobs, in bulk, sometimes at lower cost. That is batching at a larger scale: trading time for price. If your task does not need an answer immediately, such as labelling a backlog of documents, it is worth checking.
Three worked examples
A shop's FAQ assistant
A shop's assistant answers opening hours, delivery areas, return rules and payment methods. A review of a month of questions shows that about four in ten are repeats of a few dozen. The team adds an answer cache with an expiry of one day for these, and keeps the model for everything else. The hit rate is about 40 per cent, and the model bill falls by about that much. They are careful to clear the cache whenever the delivery areas change.
A policy question tool
A company lets staff ask questions about a 30-page leave policy. Every request sends the whole policy followed by the question. The same policy is sent thousands of times a day, so the team moves to a layout with the policy first and the question last, and turns on prompt caching with their provider. The question is different each time, so an answer cache would not help, but the shared start is identical, and the input cost drops.
An overnight labelling job
A team needs to tag 100,000 old support tickets by topic. Nobody needs the answers before morning. Rather than send them one at a time through the live service, they submit them in bulk overnight, using a bulk option at lower cost, and read the results at the start of the day. There is no cache to help, since the tickets are all different, but the batching is large and the price is lower.
Common mistakes
Caching answers that change. A cache with no expiry will serve stale facts.
Caching private answers for everyone. A cached answer that mentions one user must never reach another.
Putting the changing part first. Prompt caching needs the shared text at the start.
Assuming bigger batches are always better. Throughput rises, but each step takes longer and the gain flattens.
Judging batching by average wait alone. Look at the longest waits too.
Not measuring the hit rate. Without it you do not know whether the cache is helping.
Forgetting what a cache costs. It needs storage, expiry rules and care.
Using your own numbers from another system. The extra time per added request, and the provider's discount, are specific to your setup. Measure them.
Key takeaways
Batching serves several requests together to make better use of hardware, and the trade-off is throughput against waiting time.
Static and dynamic batching hold finished answers until the slowest in the batch is done; continuous batching does not.
Continuous batching usually gives the lowest wait and the highest use of the machine.
Bigger batches serve more work per unit of time, but the gain flattens and each step takes longer.
Answer caching saves the most when questions repeat and answers stay valid. Prompt caching saves the most when a long start is shared.
Cached answers need an expiry, must be cleared when facts change, and must never leak private information.
Check yourself
1. Why does static batching hold back short requests?
It returns every answer only when the slowest request in the batch finishes.
2. In the batch-size simulator, what happens to throughput and step time as you raise the batch size?
Throughput rises but flattens, while each step takes longer.
3. A cache has a 5 per cent hit rate. Is it worth building?
Probably not for general use. Only a handful of very common questions repeat, so the saving is small.
4. Why must the shared text come first in a prompt if you want prompt caching to work?
The saved work applies to the start of the prompt, so a changing part at the top breaks the match.
5. Name one rule that stops a cache from leaking private answers.
Include the user or document in the cache key, or do not cache answers that contain personal details.
Try this
Set aside about forty-five minutes and think of a feature where many requests look alike.
1. Write down a sample of twenty real or realistic requests. Count how many are repeats or near repeats. That is your rough hit rate.
2. Use the answer-cache simulator with your own volume and cost per call. Write down the monthly saving, and what expiry each kind of answer would need.
3. Write the full text of a typical request. Mark which part is the same every time and which part changes, and rewrite it with the shared part first.
4. Use the prompt-caching simulator with your own token counts. Write whether the saving is large enough to bother with.
5. List any answers that must never be shared between users, and say how you would keep them out of a shared cache.
6. Write one sentence on whether your use needs low waiting time for each user, or high total work done, and which batching style that points towards.
A good result is a one-page plan: what you would cache and for how long, how you would lay out the prompt, what you would never cache, and the rupee saving you expect with the number you are least sure of marked.
Tags: AI Engineering Batching Caching Latency Serving