Lesson 2.1: Tokens and context windows explained
Language models read tokens, not words, and can only hold so many at once. Learn what tokens are, how to estimate them, and why the context window matters.
Two ideas explain a surprising number of everyday problems with language models: why long documents get cut off, why a model seems to forget the start of a long chat, and why bills grow faster than expected. They are tokens and the context window, and this lesson makes both concrete. By the end you will be able to say what a token is, estimate how many tokens a piece of text uses, explain what the context window holds, work out whether a request will fit, and see with a simulator why a chat gets more expensive with every turn. You need nothing but arithmetic. If you have worked through the first module of this track, you already know where this fits: tokens and the window sit inside the context-building and model-call parts of the five-part map.
This is the first lesson of module two, which looks inside the model just far enough to make you a better engineer. You do not need the mathematics of how a model works. You do need to know the units it works in, because every cost, every limit and many mysterious failures are measured in them.
What a token is
A language model does not read letters or whole words. It reads tokens: small pieces of text that it has learned to treat as units. A common short word is often one token. A longer or rarer word may be split into several. Punctuation and spaces count too, and so do numbers, which are often split into pieces.
You can think of tokens as the model's alphabet, except that its "letters" are chunks chosen because they appear often in the text it was trained on. The word "the" is a single chunk. A word like "reconciliation" may be two or three. A made-up name may be split into many small pieces, close to letter by letter. This is why the model can handle words it has never seen: it breaks them into familiar parts.
As a rough rule for English, one token is about four characters, so 100 tokens is roughly 75 words. Other languages, including Hindi and other Indian languages, often take more tokens for the same meaning, because the model's vocabulary covers them less compactly. The exact numbers differ by model, so treat the rule as an estimate and use the provider's token counter when it matters.
Here is a table of rough sizes you can carry in your head. They follow the four-characters rule and are estimates, not exact counts.
Text · Approximate words · Approximate tokensOne short sentence · 15 · 20
A typical email · 150 · 200
One page of a report · 500 · 670
A 20-page report · 10,000 · 13,000
A 300-page book · 90,000 · 120,000
The figures give you a feel for scale. A model with a window of a few thousand tokens can hold a short report. A model with a much larger window can hold a book. The size of the window is therefore one of the first facts you check about a model.
The context window
A model can only look at a limited amount of text at one time. That limit is the context window, measured in tokens. It covers everything in the request and the answer the model writes back: your instructions, the chat history, any documents you include, and the reply.
Picture the window as a table of fixed size. Everything the model needs to consider has to fit on the table together: the instructions, the earlier conversation, the documents and the space for its own answer. If you put too much on the table, something has to come off.
If the total goes beyond the limit, something has to give. Either the request is rejected with an error, or older parts are dropped, and the model behaves as if it never saw them. This is why a very long conversation can lose track of an early instruction. The instruction was not forgotten the way a person forgets. It was pushed off the table.
A typical request divides the window among several things. The picture below is an illustration of one way an 8,000-token window might be used, not a rule.
Notice that the largest share goes to supplied documents and history, and that a sizeable part is held back for the answer. Teams new to this often forget the last slice, fill the window with input and then find that the model has no room to reply.
Predict first: a model has an 8,000-token window. You send a 7,500-token document and ask for a long summary. What is the problem? Decide before reading on.
Only 500 tokens are left for the answer, about 375 words, and part of that may be needed for your instructions. The summary will be cut off or the request will fail. Either shorten the input or use a model with a larger window.
Try it yourself: type or add text below and watch it fill a window.
A bigger window is not a free pass
Windows have grown a great deal, and it is tempting to think the problem has gone away: just send everything. Three things still argue for care.
Cost. You pay for every token you send, on every request. Sending a whole manual to answer a one-line question costs many times more than sending the one relevant page.
Speed. The model has to read all of it before it can answer, so longer requests are slower.
Quality. More text is not always better. Irrelevant material can distract the model, and a key instruction buried in the middle of a very long request is easier to miss than one placed with care. Experienced engineers choose what goes in, rather than sending everything and hoping.
This is the bridge to later modules: retrieval exists precisely so that you can send the few relevant pages and not the whole library.
Why tokens also mean money
Providers usually charge per token, often with different prices for tokens you send (input) and tokens the model writes (output). Suppose a request has 3,000 input tokens and 500 output tokens. If you handle 10,000 such requests a day, that is 30 million input tokens and 5 million output tokens daily. Even a small price per million tokens adds up, which is why trimming unnecessary text from requests is one of the easiest ways to cut cost.
Make that concrete with illustrative prices: suppose input costs ₹40 per million tokens and output costs ₹160 per million. Then 30 million input tokens cost ₹1,200, and 5 million output tokens cost ₹800, so the day costs ₹2,000 and a 30-day month costs ₹60,000. These prices are assumptions for illustration only, and real prices vary by provider and model, so check the current price list for the model you use. But the shape holds: output tokens usually cost more each, while input tokens are usually far more numerous. Cutting 500 unneeded input tokens from every request saves more than you might expect, because it is multiplied by every request you ever send.
The hidden cost of a conversation
There is one place where costs grow faster than people expect, and it is worth a whole section. The model does not remember a chat. Each time the user sends a message, your code sends the entire conversation so far again, plus the new message, because that is the only way the model can see the history. So a chat does not cost "one message per turn". Each turn costs more than the last, because the history keeps growing.
Work through it. Say your instructions take 400 tokens, each user message takes 80 and each reply takes 250. Turn one sends 400 plus 80, which is 480 tokens. Turn two sends 400, plus the first exchange of 330, plus the new 80, which is 810. Turn three sends 1,140. Every turn adds another 330 tokens to the request. After twelve turns the request is over 4,000 tokens, and the total input across the whole conversation is far larger than twelve times the first turn.
Use the simulator below. Set the window, the size of your instructions and messages, the number of turns and the prices. It shows how big the current request is, how much room is left for the answer, how many turns fit before the window fills, and the fees for the whole chat. The chart shows the request growing turn by turn against the window.
Try three experiments. First, leave everything and move the turns from 12 to 24. The request length doubles roughly, but the total input across the chat grows much faster, because every extra turn re-sends everything before it. Second, halve the reply size from 250 tokens to 125 and see how much that helps both the window and the bill. Third, raise the window to 128,000: the turns that fit leap, but the fees do not change, because you pay for tokens used, not for the size of the window. A bigger window removes the failure and not the cost.
How text really splits
It helps to see how unevenly text turns into tokens, because it explains why estimates drift. Plain, common English words are cheap: a sentence of everyday words uses close to one token per word, a little more with punctuation. Technical words, names and abbreviations cost more, because they are split into pieces the model has seen less often. Numbers can be surprisingly expensive: a long account number or a table of figures may use a token for each few digits. Code, with its symbols and indentation, and text with a lot of formatting, such as tables, also use more tokens than the same information written in plain sentences.
Three practical consequences follow. A page of narrative text and a page of numbers are not the same size, so measure the material you will really send. Cleaning input pays off, since stripping repeated headers, page numbers and blank space before sending can trim a large share of tokens with no loss of meaning. And when you ask for structured output, a compact format saves tokens on every response, which at volume is real money. None of this needs precision. It needs the habit of looking at the count and not assuming.
Budgeting tokens for a real feature
When you plan a feature, make a token budget the way you would make a money budget. Write down each part of the request, estimate its size and add them up, then compare with the window and with the price. For a support assistant it might look like this.
Part of the request · Tokens · NotesInstructions and rules · 500 · Sent with every request
Customer's account details · 200 · Looked up for each customer
Retrieved policy text · 1,500 · The three most relevant paragraphs
Conversation history · 1,200 · The last four exchanges
The new question · 100 · Typical length
Room kept free for the answer · 500 · Replies of about 350 words
Total · 4,000 · Fits an 8,000-token window with room to spare
Reading the budget tells you where your levers are. The retrieved text is the biggest item, so tuning how much you retrieve moves cost and quality the most. The instructions are small but repeat on every request, so trimming 200 unneeded tokens there saves 200 tokens times your daily volume. The history grows, so it needs a cap. If you later double the number of retrieved paragraphs, you can see at once that the total rises to about 5,500 and still fits, and what that adds to the bill.
A budget also gives you a number to monitor. If the average request in production is 7,000 tokens when you planned 4,000, something is leaking: perhaps the history is not being trimmed, or the documents sent are longer than expected. A token count in your logs is one of the cheapest health checks a system can have.
Strategies for long conversations and long documents
Engineers use a few standard tactics when text threatens to outgrow the window.
Trim the history. Keep only the last few exchanges plus the instructions. Cheap and often enough.
Summarise the history. Ask the model, or a cheaper one, to compress older turns into a short note, and send the note instead of the full text. You trade some detail for much more room.
Retrieve instead of paste. For documents, send only the pieces relevant to the question. This is the idea behind the retrieval lessons.
Split the work. Process a long document in sections and combine the results, for example summarising each chapter and then summarising the summaries.
Choose the window to fit the job. If most requests are short, a smaller window costs nothing extra. Pay for a large one only for the tasks that need it.
Each tactic gives something up, and the right choice depends on what matters in your product. A customer chat may need only the last few turns. A legal review may need every word and so uses splitting and careful checks.
Worked examples
Example 1: the long report
A team wants a summary of a 30,000-word annual report. By the table, 30,000 words is roughly 40,000 tokens. A model with an 8,000-token window cannot take it in one go. The team has three options: use a model with a larger window, split the report into sections of about 5,000 tokens and summarise each, or retrieve the sections most relevant to a specific question. For a general summary, splitting is common. For questions about particular topics, retrieval is cheaper and usually better.
Example 2: the assistant that forgot its rules
A bank's assistant starts a chat by being told: always remind the customer that it cannot give investment advice. After thirty turns, it begins offering opinions on funds. The team suspects the model is "getting worse". The real cause is simpler: with a small window and a trimming rule that drops the oldest turns first, the instruction at the start of the conversation was dropped along with them. The fix is to keep the instructions in every request and trim only the conversation. This is a context-building fix, which fits the five-part map from the previous module.
Example 3: the Hindi support bot
A support bot answers in English and Hindi. The team estimates costs from English tests and is surprised when the Hindi conversations cost more per message. Hindi text often needs more tokens than the English text of the same meaning, so the same reply uses more of the window and costs more. The lesson is to measure on the language and the content you will really have, and to use the provider's token counter and not a rule of thumb.
Common mistakes
These mistakes cause most of the surprises in the first months of working with models.
Counting words instead of tokens. Limits and prices use tokens. Estimate with the four-character rule and check with a counter.
Forgetting the answer takes space. The reply counts against the window. Keep room for it.
Thinking the model remembers. It sees only what you send each time. History must be re-sent, and it costs tokens.
Letting history grow without limit. Trim or summarise, or the bill and the failures will grow with every turn.
Sending everything "just in case". Extra text costs money, slows the answer and can distract the model.
Assuming a bigger window fixes everything. It removes one failure, but you still pay for every token and the model still works best with relevant text.
Estimating from English only. Other languages can cost more tokens for the same meaning.
Key takeaways
Models read tokens, roughly four English characters each, not words, and other languages may need more.
The context window is the total tokens the model can handle at once, including the answer.
Text beyond the window is rejected or dropped, so the model can seem to forget.
Providers charge per token, so shorter, more relevant requests cost less.
A conversation re-sends its history every turn, so cost grows faster than the number of turns.
A bigger window prevents failures but not cost, and more text is not always better text.
Check yourself
1. Roughly how many tokens is a 600-word email?
About 800, because 100 tokens is roughly 75 words, so 600 divided by 0.75 is 800.
2. A window is 4,000 tokens and the input is 3,800. Why might the answer be cut off?
Only 200 tokens remain for the reply, since the answer counts against the same window.
3. Why does a 20-turn chat cost more than twice a 10-turn chat?
Each turn re-sends the whole history, so later turns carry more tokens and the total grows faster than the number of turns.
4. Instructions take 400 tokens, and each exchange adds 330. About how many tokens is the request on turn 10, with a new 80-token message?
About 3,450: 400 plus 9 times 330 plus 80.
5. Does moving to a larger window lower the cost of a chat?
No. You pay for the tokens you send, so a larger window avoids failures but does not cut the bill.
Try this
Set aside about forty minutes and pick a text you use at work, such as a report, a policy or a long email thread.
1. Count its characters, divide by four and write down the token estimate. Then paste it into the counter above and compare.
2. Work out how many pages like it would fit in an 8,000-token window while leaving 1,000 tokens for an answer.
3. Imagine a chat about it with 400 tokens of instructions and typical messages. Put your guesses for message and reply sizes into the simulator and find how many turns fit.
4. Find the total fees for one chat and for a day's worth at a volume you think realistic, using prices from a provider you might use.
5. Choose a tactic from the section above, trimming, summarising, retrieving or splitting, and write one sentence on why it suits your text.
6. Write down what you would lose by using that tactic.
A good result is a short note that gives the token size of your text, the number of turns or pages that fit, the cost at your volume and the tactic you would choose. It is the first piece of arithmetic any real project needs.
Tags: AI Engineering LLM basics Tokens Context window Beginner