Lesson 1.2: The parts of an LLM application
Almost every language model product has five parts: input, context, model, output handling and feedback. Learn the map so every later lesson has a place.
A chat window can make a language model product look like magic, but underneath it is a fairly ordinary system with a few well-defined parts. Once you can name those parts, every technique in this track, from embeddings to evaluation, has an obvious place to sit, and a wrong answer stops being a mystery and becomes a question you can answer: which part failed? By the end of this lesson you will be able to draw the five parts of a language model application from memory, say what each one does and what can go wrong in it, follow one request through all five, and use a simple calculation to see why a system made of decent parts can still give wrong answers often.
You do not need to have built anything yet. If you finished the first lesson, you already know that the model is a component and the engineering is the system around it. This lesson draws the plan of that system.
Why a map helps
Beginners often treat an AI application as a single thing: you type, it answers. When the answer is wrong, the only knob they can see is the wording of the question, so they spend hours rephrasing. Experienced engineers see a pipeline. They ask where in the pipeline the mistake entered, because the fix depends on the answer. A wrong refund policy might need better documents, a different instruction, a stricter output check or a smarter model, and those are four different pieces of work.
A map also helps you talk to colleagues. When a product manager says "the assistant is wrong sometimes", you can reply "which kind: did it get the wrong text, did it misread the text, or did we show its answer without checking?" That one question can turn a vague complaint into a plan.
The five parts
Follow one request from a user to an answer.
1. Input. What the user typed, spoke, uploaded or triggered. It may be a question, a form, a voice message or a file.
2. Context building. Your code decides what else the model needs to see: instructions, earlier messages, relevant documents, the user's account details. This is assembled into a single request.
3. The model call. The request goes to a language model, which returns text.
4. Output handling. Your code checks and shapes the text: it might parse it into fields, hide unsafe content, or run an action the model asked for.
5. Feedback and logging. You record what happened, so that you can measure quality and fix problems later.
The model call is one step in five. Most of the engineering effort sits around it. Step through the example below to watch one request travel through all five parts.
Now take each part in turn, with what it involves and where it tends to go wrong.
Part 1: input
The input looks like the simplest part, and it is not. Real users do not write clean questions. They type "refund??", paste a whole email thread, upload a blurry photograph of a bill or speak into a phone on a noisy road. Your system has to turn whatever arrives into something the rest can use.
For a voice product, input includes speech recognition, and a mistake there flows through everything after it. If the caller says "forty thousand" and the transcript says "fourteen thousand", the best model in the world will answer the wrong question. For a document product, input includes reading the file, and a scanned page that comes out as scrambled text has the same effect.
Typical design questions in this part are how to handle empty or very long input, how to clean text without losing meaning, how to detect the language, and how to refuse requests you do not want to handle, such as anything outside the product's purpose.
Part 2: context building
Context building is where the most important work happens, and it is the part newcomers underestimate. A language model has no memory of your company, your user or your last conversation. Everything it knows about the task, beyond what it learned in training, has to be in the request. Your code is the one that puts it there.
A typical request is assembled from several pieces.
Instructions that say what role to play, what to do and what not to do.
The conversation so far, because the model does not remember earlier turns unless you send them again.
Retrieved documents relevant to the question, such as the right paragraphs of a policy.
Facts about the user, such as their plan, their account type or the language they prefer.
The question itself.
Deciding what to include is a real design problem, because space is limited and everything you add costs money and time. Include too little and the model guesses. Include too much and it may lose the thread, and the bill grows. Later modules cover how to find the right documents and how to fit them in.
Part 3: the model call
The model call is the step everybody pictures, and it is the one you control least. You send the assembled request and get text back. You can choose which model to use, and you can adjust settings such as how adventurous the wording should be, but you cannot look inside. What you can do is make the request clear and the settings sensible, and then treat what comes back as a draft that needs checking.
Two facts about this step shape the rest of the design. First, a model can be wrong while sounding certain: fluent text is not evidence of correct text. Second, the same request can produce different answers on different runs, so you cannot test a system by trying one example once.
Part 4: output handling
Output handling is the code that stands between the model's text and the rest of the world. Its jobs depend on the product.
Turn text into data. If the application needs a category, a date or an amount, the code parses the model's reply and checks that it is valid.
Check the answer. Does it quote the policy? Is the number within a sensible range? Does the total add up?
Remove what should not be shown. Personal details, unsafe content or anything that breaks a rule.
Take actions. If the model asked to look up an order or send a message, the code decides whether to do it.
Choose a fallback. If the output fails a check, show a safe message, try again or hand over to a person.
Good output handling is what makes a product trustworthy. A system that shows the model's raw text to customers is only as reliable as the model on its worst day.
Part 5: feedback and logging
The last part does not affect the answer in front of the user. It affects every later answer. Logging means saving what was sent and received, along with what happened next: whether the user clicked "helpful", whether a person corrected it, how long it took and what it cost.
Without logs you cannot tell whether a change improved things, you cannot reproduce a complaint and you cannot find patterns in the failures. With logs you can pull out the hundred worst conversations and read them, which is one of the most valuable things a team can do. The one rule is to treat logs as sensitive data, because they contain what users typed. Later lessons cover privacy.
What goes wrong where
When something fails, the five parts give you a checklist. This table shows common failures and where to look. Keep it handy, because it will be useful in nearly every later lesson.
Symptom · Most likely part · What to check firstThe answer is about the wrong thing · Input · Did the system understand the question, or mishear it?
The answer is confident but uses outdated or wrong facts · Context building · What documents and instructions were actually sent?
The answer ignores an instruction · Model call · Is the instruction clear, and is it buried in a long request?
The answer is right but shown in the wrong format · Output handling · Is the parser too strict, or the output rule missing?
The same mistake keeps happening and nobody noticed · Feedback and logging · Are you logging and reviewing real conversations?
The pattern in this table is the most useful idea in the lesson: when an answer is wrong, check the context first. Surveys of why answers go wrong will differ between products, but the common finding is that most bad answers trace back to what the model was shown, not to the model's ability. Here is an illustration of what a review of 100 bad answers might find. The numbers are made up for teaching, to show the shape, and your own results will differ.
If you only changed the model in this example, you would be working on the second bar and hoping it moved the others. The largest bar is context, and fixing it needs different work: finding better documents, keeping them up to date and sending the right pieces.
Predict first: a support assistant tells a customer the refund window is 30 days when the policy says 14. Which part most likely failed: the model call, or something earlier? Decide before reading on.
Most often it is context building: the right policy paragraph never reached the model, or an outdated one did. The model answered confidently from what it was shown. Blaming the model first sends you in the wrong direction, so check the context first.
Why five good parts can still give wrong answers
There is a second lesson hidden in the pipeline, and it is a matter of arithmetic. Every request must pass through the parts in order, and a mistake in any one can spoil the answer. When several things must all go right, the chance that all of them do is the product of the individual chances, and products shrink quickly.
Suppose each part works well on its own: the input is understood correctly 98 times in 100, the right context reaches the model 85 times in 100, the model answers correctly given good context 95 times in 100 and output handling passes a correct answer through properly 98 times in 100. Each looks decent. Multiply them and the whole chain gets a correct answer only about 78 times in 100. Put another way, about one request in five is wrong, and no single part looks terrible.
Use the simulator below to see this. Move each slider and watch the whole chain's reliability. Then try improving only the weakest part and notice how much more that gains than polishing a part that is already strong.
Two experiments are worth running. First, with the starting numbers, raise the model's score from 95 to 100 per cent and see how little the chain improves, because the model was not the bottleneck. Then lower context from 85 to 70 and watch the whole chain drop sharply. The second line on the chart answers a useful question: even a perfect model cannot lift the chain above the level set by the other parts. The model's quality matters, and the system's quality is set by its weakest link.
A worked example: the leave policy tool
Take a tool that answers questions about a company's leave policy, and follow one request.
Input: "How many casual leaves do I get?"
Context building: add an instruction ("Answer only from the policy text"), the user's grade, and the two policy paragraphs most related to leave.
Model call: send everything and receive a short answer.
Output handling: check the answer includes a quote from the policy, and hide it if not.
Logging: save the question, the paragraphs used and whether the user clicked "helpful".
If users later report wrong answers, the log shows which paragraphs were sent, so you can tell in minutes whether the retrieval or the model was at fault. Without that log, the team would only know that "the assistant sometimes gets leave wrong", and the only way to investigate would be to try questions again and hope to see the same mistake.
A second example: a voice assistant for loan queries
Now take a phone assistant for a lender that answers questions about loan balances. The five parts are the same, with different details.
Input is speech turned into text. The caller says "what is my outstanding on the personal loan?" and the system must recognise the words, in the caller's accent and over a noisy line, and also work out who the caller is.
Context building looks up the caller's real balance from the lender's own system and adds it, along with an instruction about tone and what the assistant may and may not disclose.
The model call writes a short, polite reply using those facts.
Output handling checks that the amount in the reply matches the amount that was looked up, because a wrong figure in a loan conversation is serious. It also keeps the reply short enough to be spoken.
Logging records the call, with care, since it contains personal and financial details.
Notice that the most important fact in this system, the balance, never comes from the model at all. It comes from the lender's database, put into the request by context building and checked afterwards by output handling. The model only phrases it. That is a common and good pattern: let the model do language, and let ordinary systems supply and verify the facts.
A third example: the refund assistant
Return to the refund-window mistake. An online shop's assistant told a customer 30 days, when the policy says 14. The team reads the log and finds the request included a paragraph from an old policy page, because the document collection held both versions. The model quoted the old one faithfully. The fix is in context building: remove the outdated page, and add a date to each document so that newer text wins. A check in output handling can add safety, for example by requiring that any refund window quoted in an answer match the current policy table.
The point of the example is the order of investigation. The team did not start by changing the model or rewriting the instruction. They started by reading what was sent, and the answer was in the log.
Where the later lessons fit
The rest of the track maps onto these five parts.
Writing good instructions and handling long text belongs to context building.
Embeddings and retrieval are a way to find the right documents for context building.
Choosing and configuring a model belongs to the model call.
Structured outputs and guardrails belong to output handling.
Evaluation and monitoring belong to feedback and logging.
Agents repeat steps two to four in a loop, letting the model take several actions before answering.
Keep this picture in mind. When something goes wrong, you can ask which part failed, and each answer points to a module of this track.
Common mistakes
These are the errors that most often mislead people building their first application.
Rewriting the prompt when the context is wrong. If the right text never reached the model, no wording will help. Read what was actually sent.
Not logging what was sent. Without it you cannot tell which part failed, and you will argue about the model instead.
Showing raw model output to users. Always put a check between the model and the user, even a simple one.
Letting the model supply facts it should look up. Balances, dates and prices belong to your systems. The model should phrase them, not remember them.
Assuming the model remembers earlier turns. It does not unless you send them again.
Testing with one example. The same request can give different answers, so test across many.
Improving the strong part. Polishing a model that is already good, while the context is poor, moves little. Find the weakest link first.
Key takeaways
An LLM application has five parts: input, context building, the model call, output handling, and feedback and logging.
Most of the engineering effort is around the model call, not inside it.
When an answer is wrong, check the context first: what was actually sent, and was it right and current?
Reliability multiplies down the chain, so several decent parts can still give wrong answers often, and the weakest part matters most.
Let ordinary systems supply and verify facts, and let the model do the language.
Logging what was sent and received is what makes debugging possible.
Check yourself
1. Name the five parts of an LLM application in order.
Input, context building, the model call, output handling, and feedback and logging.
2. A bot quotes a discontinued offer. Which part do you look at first, and why?
Context building: the old offer was probably in the documents sent, and the log will show it.
3. Why is it risky to show the model's raw text to users?
The model can be wrong while sounding certain, so a check between it and the user catches errors.
4. Four steps each work 90 per cent of the time. Roughly how often is the whole chain right?
About 66 per cent, because 0.9 multiplied four times is 0.656.
5. In the loan balance example, why is the balance not taken from the model?
The balance is a fact held by the lender's system. The model should phrase it, and output handling should check that the figure matches.
Try this
Set aside about forty minutes and choose an AI feature you have used, such as a banking assistant, a shopping helper or a writing tool.
1. Draw the five parts on one page and write what you think goes in each for that feature.
2. Mark the part you think would be hardest to get right, and say why.
3. Write down three ways the feature could give a wrong answer, and for each one name the part that would have failed.
4. Guess a probability for each of the four answer-path parts, and use the reliability simulator above to see how often the whole chain would be right.
5. Find the weakest part in your guesses and describe one change that would improve it.
6. Decide what you would want to log to find out whether your guesses were right.
A good result is a one-page diagram with the five parts, three failure stories assigned to parts and one proposed improvement. It is the first draft of the design notes an engineer writes before building.
Tags: AI Engineering LLM applications System design Architecture Beginner