Lesson 3.3: Streaming, latency and cost of a request
Every model request has a price and a wait. Use the sliders to see how input size, output size and volume change your bill and what the user feels.
Two numbers decide whether an AI feature is practical: what it costs to run and how long the user waits. Both come from the size of each request, and both can surprise you at scale. By the end of this lesson you will be able to estimate the monthly cost of a feature from a handful of numbers, explain why a request has two phases of waiting, say when streaming helps and when it does not, see why long conversations cost far more than their length suggests, and judge when sending easy requests to a cheaper model is worth it. All the prices and speeds here are illustrative assumptions, set out in the sliders so you can replace them with your provider's real figures.
Where the cost comes from
Providers usually charge for tokens, with separate prices for the tokens you send (input) and the tokens the model writes (output). Output tokens commonly cost more per token than input tokens. The monthly bill is roughly:
requests per day × 30 × (input tokens × input price + output tokens × output price)
Because long documents inflate the input and long answers inflate the output, the same feature can cost five times more with a small change in how much text you send.
Here is the arithmetic for a worked example that we will use throughout. Suppose a feature gets 20,000 requests a day. Each request sends 2,000 tokens and gets back 400. Take an illustrative input price of ₹40 per million tokens and an output price of ₹160 per million tokens, which means output is four times as expensive per token. In a month there are 600,000 requests. The input is 1,200 million tokens, costing ₹48,000. The output is 240 million tokens, costing ₹38,400. The total is ₹86,400 a month.
Notice something that surprises many people. The output is only a fifth of the tokens, yet it is nearly half of the bill, because each output token costs more. When you look for savings, check both sides, and work out which one is bigger in your case.
Where the wait comes from
A request has two phases. First the model reads your input, and then it writes the answer one token at a time. The wait before the first word depends mostly on the input size. The time to finish depends mostly on the output length.
The reading phase is quick because the model can process the whole input at once. The writing phase is slow because each new token depends on all the ones before it, so they come out one after another. With the illustrative speeds used in this lesson, which are 4,000 tokens a second for reading and 50 tokens a second for writing, plus a short fixed delay of 0.4 seconds, the example request takes 0.4 plus 0.5 seconds to read 2,000 tokens, which is 0.9 seconds before the first word. Writing 400 tokens then takes another 8 seconds, so the whole answer takes about 8.9 seconds.
Streaming sends tokens to the user as they are produced, instead of holding the whole answer until it is done. The total time is the same, but the user starts reading after the first phase instead of after both. For a long answer, that can turn a nine-second blank screen into a one-second wait.
Try it with the sliders below. The rates are assumptions chosen for illustration, listed under the chart.
Predict first: you double the input tokens but leave everything else the same. Does the monthly cost double? Decide, then try it.
Not quite. Only the input part doubles, so the total rises by less than double, and it depends on how much of your bill was input. With the starting values, halving the input saves about ₹24,000 a month and halving the answer saves about ₹19,200, so which one to trim first depends on your own numbers. Try it.
A worked example
With the starting values (2,000 input tokens, 400 output tokens, 20,000 requests a day) the assumed rates give about ₹86,400 a month, and a request takes about 8.9 seconds to complete. With streaming the user sees the first words after about 0.9 seconds. Cutting the average answer from 400 to 200 tokens saves roughly ₹19,200 a month and almost halves the writing time, which is why asking for short, structured answers is a cheap and effective fix.
When streaming helps, and when it does not
Streaming is one of the cheapest improvements to how a feature feels. It does not make the model faster. It changes when the user starts getting something. A person reading a streamed answer begins reading after the first second or so and, if they read at a normal speed, is often still reading when the last words arrive. They experience little or no waiting at all.
But streaming is not always the right choice.
Use it when a person reads the answer live, such as in a chat window or a drafting tool.
Do not use it when a program needs the complete answer before it can act, such as a system that reads a structured result and passes it on. A half-finished answer is useless to a program, and handling partial output adds complexity.
Be careful with answers that you check before showing, such as filtering for unsafe content. If you stream, the user may see words you would have removed. Some products stream and then correct; others hold the answer until it has been checked.
Streaming also changes what you should measure. The number that matters to the user is the time to first word, and the number that matters for cost is the total tokens. Track both.
Long conversations cost more than they look
The cost per request in our example looked simple, but a chat is not one request. Each time a user sends a new message, the application usually sends the whole conversation so far again, because the model does not remember earlier turns by itself. The system instructions, every earlier question and every earlier answer all go in as input, and then the new message is added. As the chat gets longer, each turn carries more input than the last.
That means the cost of a conversation grows faster than its length. Ten turns do not cost ten times as much as one turn. Use the simulator below to see this. It adds up the input and output tokens across a whole conversation, using prices you can change.
At the starting numbers, a ten-turn conversation costs about ₹1.18, and the last turn sends 3,350 tokens. Double the turns to twenty and the cost rises to about ₹3.60, roughly three times as much, not twice. The curve bends upward because the history grows with every turn. This is why long conversations are a common source of a surprisingly large bill, and why products often shorten or summarise the history after a certain length, a technique later lessons cover.
Levers for cost and delay
There are only a few things you can change, and each affects cost and wait differently. This table puts them side by side.
Lever · Effect on cost · Effect on wait · CautionShorter answers · Cuts output cost directly · Cuts the writing time most · Ask for the length you need, so you do not lose useful detail
Less input · Cuts input cost · Cuts time to first word · Do not drop context the model needs
Streaming · None · Makes it feel faster, no change to total · Not suited to programs that need the full answer
A smaller, cheaper model for easy tasks · Can cut cost a lot · Often faster too · Quality may fall on harder requests
Caching repeat work · Cuts both · Cuts both · Needs expiry and care with private data
Limiting conversation length · Cuts the history effect · Keeps late turns quick · Users may lose context
A good habit is to decide which of the two matters more for your feature. A tool used by staff to draft long documents may care about cost and tolerate a wait. A voice assistant answering a customer on the phone cares a great deal about the first second, and may accept a higher cost per request.
Sending easy requests to a cheaper model
One of the most effective levers is not to use the biggest model for everything. Many requests are simple, such as short factual questions or classification, and a smaller, cheaper model may handle them well. Harder requests can go to the larger model. Choosing which request goes where is called routing.
The saving depends on what share of requests the cheaper model can handle, and how much cheaper it is. The simulator below uses the example from earlier and lets you set both.
With 60 per cent of requests going to a model that costs a fifth as much, the monthly cost falls from about ₹86,400 to about ₹44,900, a saving of roughly ₹41,500. The line on the chart falls steadily as more requests move across. The catch is the one in the caption: the saving is real only if the cheaper model's answers are good enough. A common approach is to test the cheaper model on a sample of real requests, choose the kinds it handles well, and keep a way for a hard request to fall back to the larger model.
A five-step estimate before you build
You do not need a working system to estimate cost and wait. A rough calculation on paper, done before any code is written, is often enough to tell you whether an idea is viable, or which part to design carefully. Here is a method you can repeat for any feature.
1. Write a typical request. Take one realistic example, with the instructions you would send, any documents or history, and the user's message. Paste it into a token counter, or estimate it as roughly three quarters of a word per token, and write down the input size.
2. Write a typical answer. Decide how long a good answer should be, and note the output size. If you are not sure, write the answer you would want by hand and count it.
3. Estimate the volume. How many requests a day, and how many turns if it is a chat? Use a low, an expected and a high case, because early guesses are often wrong by a large factor.
4. Do the arithmetic. Multiply the tokens by the prices, and by the volume, as in the formula above. Work out the time to first word and the total time from the speeds. Do it for the three volume cases.
5. Look for the biggest term. Is it the input, the output or the history? Is the first word fast enough? That is where to spend design effort.
Suppose a team plans a tool that helps sales staff draft follow-up emails. A typical request has 800 tokens of instructions and notes, and the answer is about 250 tokens. They expect 300 requests a day to start, and a hundred times that if it succeeds. At the example prices the input costs about ₹0.032 a request and the output about ₹0.04, around ₹0.07 in all. At 300 requests a day that is a few hundred rupees a month, which is nothing. At 30,000 requests a day it is about ₹65,000 a month, which needs a decision. The team learns two useful things: the cost is no obstacle to a pilot, and it will matter if the tool takes off, so they log tokens from the start.
A similar calculation helps with the wait. With the example speeds, 800 input tokens take 0.2 seconds to read, and 250 output tokens take 5 seconds to write, so the draft is complete after about 5.6 seconds. Streamed, the first words appear after about 0.6 seconds. If the team decided a person needs to see something within a second, the design passes with streaming and fails without it.
The value of this exercise is not the exact numbers, which depend on your provider. It is the habit of knowing which number matters before you build, and of noticing early when a feature that looks cheap in a demo will not be cheap at scale.
Keeping an eye on the bill
Costs creep. A new feature, a longer instruction, a user who pastes a long document: each can move the bill without anyone deciding to. A few habits keep it under control.
Log tokens for each request, split into input and output, and note which feature it came from. Without this you cannot tell what is driving the bill.
Set a budget and an alert. A daily spend limit with a warning at, say, eighty per cent catches runaway usage early.
Cap what users can send. A maximum input length, and a maximum answer length, put a ceiling on the worst case.
Review the biggest requests. Looking at the top few per cent by cost often reveals one prompt or one user responsible for much of the bill.
Three worked examples
A customer-support chat
A support chat has short questions and short answers, but conversations run to fifteen or twenty turns. The team's first estimate multiplied one turn by the number of turns, and the real bill came in at about three times that, because of the history effect. They respond by shortening the system instructions and summarising the older turns after ten turns. Time to first word stays low because they stream.
A report summariser
An analyst tool takes a long report as input, around 20,000 tokens, and returns a short summary. Here the input dominates, so trimming the answer saves almost nothing. The team looks instead at how much of the report is really needed, extracts the key sections first with a cheaper model, and sends only those to the larger one.
A form-filling assistant
A program reads a document and returns a small structured result that another system uses. Nobody reads the answer as it arrives, so streaming brings no benefit, and it adds complexity. The team skips it and focuses on keeping the answer short and the format strict, which cuts both cost and time.
Common mistakes
Multiplying one turn by the number of turns. Chat history makes later turns cost more.
Looking at only the input, or only the output. Check both, and note which is larger.
Assuming streaming makes the model faster. It changes when the user sees words, not the total time.
Streaming to a program. A half-finished answer is no use to code that needs the whole.
Using the biggest model for everything. Many requests do not need it, but test before you route.
No limits on input length. One very long paste can cost more than a thousand normal requests.
Not logging tokens. If you cannot see where the tokens go, you cannot cut them.
Using example prices as real ones. Check your provider's current prices and speeds.
Practical habits
Estimate cost per request first, then multiply by expected volume.
Stream anything a person reads live, such as chat answers.
Do not stream when a program needs the complete answer before it can act.
Ask for the length and format you need, and no more.
Send only the context that helps.
Log tokens per feature, and set a budget with an alert.
Key takeaways
Cost is tokens in plus tokens out, usually priced separately, multiplied by volume.
The wait before the first word depends mostly on input size; the rest depends on output length.
Streaming improves how fast an answer feels, not how long it takes.
Chat history is re-sent each turn, so conversation cost grows faster than its length.
Shortening answers and trimming input are the quickest ways to cut both cost and delay.
Routing easy requests to a cheaper model can save a lot, but only if quality holds on those requests.
Check yourself
1. In the example, output is a fifth of the tokens but nearly half the bill. Why?
Each output token costs more than an input token, four times as much at the example prices.
2. What does streaming change, and what does it leave the same?
It changes when the user starts seeing words. The total time to finish stays the same.
3. Why does a twenty-turn chat cost roughly three times as much as a ten-turn one in the simulator, not twice?
Each turn re-sends the growing history, so later turns carry far more input.
4. When would you choose not to stream?
When a program needs the complete answer before it can act, or when answers must be checked first.
5. Name one risk of routing requests to a cheaper model.
Its answers may not be good enough for some requests, so quality can fall unless you test and keep a fallback.
Try this
Set aside about forty-five minutes and pick a feature you might build.
1. Estimate the input and output tokens of a typical request and the daily volume. If it is a chat, estimate how many turns a typical conversation has.
2. Use the cost and wait sliders above to find the monthly cost and the time to first word, with and without streaming.
3. Use the conversation simulator with your own turn count and token sizes. Write down the cost of one conversation, and what it would be at twice the turns.
4. List three requests your feature gets that look easy. Use the routing simulator with a plausible share and price ratio, and write what it would save.
5. Decide whether cost or wait matters more for your feature, and name the single change that would help most.
6. Write the three numbers you would log for every request, and the spend at which you would want an alert.
A good result is a half-page note: the estimated bill, the time to first word, the biggest lever you would pull first, and the number you are least sure of.
Tags: AI Engineering LLM APIs Latency Cost Streaming