Lesson 4.3: Building with language models in a customer setting
A demo that wows is not a system a customer can rely on. Learn to start simple, ground answers, test on real cases and keep a person in the loop.
By the end of this lesson you will be able to design a language-model feature for a customer so that it is useful and safe to rely on. You will learn a ladder of designs from a single prompt to an agent and why to climb it slowly, five ingredients of a dependable assistant, a way to test it on real cases and to say honestly how accurate it is, and three ways of keeping a person in control. You will see three customer projects, one of which went wrong in an instructive way, and use a simulator to see how much a small test set can and cannot tell you. This lesson follows 4.2, where you decided what to build. Here you build the part that many customers now ask for first, a feature powered by a language model. You should have a general idea of what a language model is. If you want more depth on how the models work, the AI Engineering track on this site covers it, and this lesson concentrates on what changes when you deploy one in a customer's world.
Language models are the easiest technology to demonstrate and the hardest to depend on. A demo takes an afternoon, and it can look astonishing: the assistant reads a document, answers a question, drafts a reply. Then the system meets the customer's real documents, real questions and real stakes, and the wrongness that was invisible in the demo appears. It states an old refund rule with complete confidence. It misreads a field in a scanned form. It answers a question that should have been passed to a person. The skill of a forward deployed engineer is to close the gap between the demo and something a business can trust, and that is mostly engineering and judgement, not clever prompts.
Why a customer setting is different
Three things change when you move from a personal experiment to a customer deployment.
The stakes are real. A wrong answer might mean a wrong charge, a wrong medical instruction or a message to a customer that the company will be held to. The cost of an error is not "try again". It is money, trust and sometimes legal exposure.
The data is messy and private. Customers' documents are inconsistent, out of date and full of personal information. What you send to a model, and where it goes, becomes a security and compliance question, as Lesson 3.4 described.
Someone is accountable. When the model is wrong, the customer, not the model, answers to their customers and regulators. They will want to know how you decided it was good enough, and "it seemed fine" is not an answer.
These facts shape every decision below. They are why the field work is different from the demo work, and why a good deployment looks, to a newcomer, surprisingly boring: narrow tasks, plain structure, checks everywhere and a person in the loop.
Start simple, and climb only when you must
A useful way to think about designs is as a ladder, from simple to complex. Anthropic's engineering guidance on building with language models is blunt about this: find the simplest solution possible, and add complexity only when it clearly improves results. A single model call with retrieval and good examples is often enough. The rungs, from the bottom up, are these.
1. One prompt. You give the model an instruction and an input, and it returns an output. Simple, cheap, fast, and often enough for classification, summaries and drafting.
2. A prompt with the customer's documents. You retrieve the relevant pieces of the customer's own material, such as help pages or policies, and give them to the model with the question, so that it answers from them and not from memory. This is called retrieval-augmented generation. It is the most common step up, and it matters a great deal for accuracy.
3. A chain of steps. You break the task into stages, each a separate call, with checks between them: first extract the facts, then check them against a rule, then draft the reply. It trades some speed for accuracy and makes each step easier to test.
4. Routing. You first classify the request and send it to the right handler: a simple question to a cheap, fast path, a sensitive one to a person. This saves cost and keeps hard cases out of automated paths.
5. An agent. The model decides its own sequence of steps and which tools to use, looping until it finishes. Agents suit open-ended problems where you cannot fix the steps in advance. They cost more, take longer and can compound their own errors, so they need guardrails such as a limit on the number of steps and human checkpoints.
The advice is to start at the bottom. Most customer tasks, such as drafting a reply, extracting fields from a form or summarising a case, are well defined and do not need an agent. A fixed workflow is easier to test, cheaper to run and far easier to explain to a security team. Move up a rung only when you can show, with test results, that the simpler design is not good enough.
The same guidance warns about frameworks. They can help you start but may hide the prompts and responses that you most need to see when something goes wrong. Begin close to the model's interface, and understand what any tool does before you rely on it.
Five ingredients of a dependable assistant
Whatever rung you choose, five ingredients make the difference between a demo and a tool.
1. A narrow, well-defined task
"Answer customer questions" is not a task. "Draft a reply to a question about the due date or the charges on a statement, using the help pages, for an agent to review" is. The narrower the job, the easier it is to test, to constrain and to explain. If the customer wants breadth, build it as several narrow tasks behind a router.
2. Grounding in the customer's own material
Models answer from patterns in their training, which are general, sometimes out of date and often confidently wrong about specifics. For a customer's facts, such as a refund policy, a product price or a medical protocol, supply the facts in the prompt, taken from the customer's current documents, and instruct the model to answer only from them. Ask it to quote or point to the source passage, and to say "I do not know" when the material does not cover the question. Show the source to the person who reads the answer, so they can check it in seconds.
Grounding moves the problem. Instead of "is the model right?", you ask "is the source right, and did we retrieve the right piece?" Both are easier to test. It also means the quality of the customer's documents matters, and a data-quality check on the source material, as in Lesson 3.3, is part of the job.
3. Structured, checkable outputs
Where the output feeds another system, ask for a fixed structure, such as a set of named fields, and check it with ordinary code. A form-reading assistant that returns an amount, a date and a name can be checked: is the date a real date, is the amount within a plausible range, does the name match the account? Ordinary rules catch many errors for free. The model is good at reading messy input. Your code is good at being strict.
4. Guardrails
Decide what the system must never do, and enforce it outside the model. Do not rely on a sentence in the prompt to stop a refund promise. Use code that detects a refund promise and sends the draft to a person. Typical guardrails include topics that always go to a human, a list of words or amounts that trigger review, a limit on the number of steps an agent may take and a check that private data is not repeated where it should not be.
5. A person in control
For almost every customer task, a person should be able to see, correct and override the model's output, at least at first. The next sections return to this.
Testing on real cases
You cannot judge a language model feature by trying it a few times and being impressed. The only reliable method is to test it on a set of real cases with known good answers, and to measure.
Build a test set early, with the customer. Collect fifty to a few hundred real examples: real questions, real forms, real cases, spread across the common types and including the awkward ones. For each, write down what a good answer looks like, with the customer's expert. This takes effort, and it is the most valuable thing you will do. It gives you three things.
A number. The share of cases handled correctly, which you can state and defend.
A list of errors. Reading the failures shows what kinds of mistake the system makes: a wrong fact, an out-of-date source, a missing caveat. Each kind has a different fix.
A safety net for change. Each time you change the prompt, the documents or the model, run the test set again. If the score falls, you know before the customer does.
Be careful about what the number means. A result on fifty cases is an estimate with a margin around it, and the simulator below makes that margin visible. Be equally careful about claiming too much. "The assistant is 94 per cent accurate" means nothing in general. "On our test of fifty real statement questions, it was fully correct on forty-six, with the four errors as follows" is a statement you can stand behind.
Predict first: the sponsor asks, "How accurate is the assistant?" You have a test of fifty real cases on which it was right in forty-six. How do you answer? Think about what you would say, and what you would not say, before reading on.
A good answer states the test and its limits: "On fifty real questions from your own history it was fully correct on forty-six, which is 92 per cent. With a test that small, the true figure could plausibly be anywhere from the low eighties to the high nineties. The four errors were two wrong facts, one out-of-date source and one missing warning. I would like to extend the test to two hundred cases before we widen the pilot, and keep an agent reviewing every draft until we do." Do not say "it is 92 per cent accurate" as if it were a fixed property, and do not say "it is very accurate". The first overclaims, and the second says nothing. Honesty about uncertainty is also what makes customers trust the number you do give.
How sure can a small test make you?
The more cases you test, the narrower the uncertainty about the true accuracy. A rough rule is that the margin around a measured proportion shrinks with the square root of the number of cases. With a small test, a measured 92 per cent leaves a wide range of plausible values. With a larger one, it narrows.
The simulator below shows the margin. Set the number of test cases and the measured accuracy. It gives the margin of error, the plausible range, and the number of errors per thousand that the lower end of that range would mean. It also plots how the margin shrinks with more cases. The calculation is a standard approximation, and it assumes your test cases resemble the real ones, which is the part you control by choosing them honestly.
Try three experiments. First, keep the defaults and read the range for fifty cases. Second, raise the number of cases to 400 and see the range narrow. Third, set the measured accuracy to 99 per cent with fifty cases and notice that even a near-perfect score on a small test leaves room for a much worse real figure.
At the defaults, 46 of 50 correct gives a margin of about 7.5 points. The plausible range runs from about 84.5 per cent to the top of the scale, which could mean 155 errors in every thousand cases. At 400 cases and the same measured accuracy, the margin falls to about 2.7 points. This is why, in a real project, a first test of fifty cases is a start and a larger one is a requirement before the system is relied upon for something that matters.
Keeping a person in control
How much a person should check depends on how much a mistake costs. Three patterns cover most deployments. Choose among them case by case, and agree the choice with the customer.
Draft only. The model prepares a draft, and a person reads, edits and sends it. Nothing reaches a customer unseen. This is the safest pattern and the right starting point for most tasks that touch customers. Its weakness is that people tire and begin to approve without reading, so supply the source passage next to the draft and audit a sample of approved items to see whether reviewers still catch errors.
Review by exception. The system handles routine, low-risk cases automatically and sends the rest to a person. The decision uses signals such as the type of request, the amount at stake, a missing source, or the model's own flagged uncertainty. It saves time, but only works if the routing is itself tested. Start by sending nearly everything to review, and relax it as the evidence grows.
Automatic with audit. For genuinely low-stakes work, such as tagging internal tickets, the system acts alone and a person samples its decisions afterwards. Use it only when an error is cheap and easily undone, and keep the sampling going.
Whatever the pattern, keep a record of what the model produced, from which sources and what the person did with it. This log is how you find the cause of a failure, how you show the customer how well it works and how you answer an auditor.
Where the data goes
Using a language model usually means sending some of the customer's data to a model provider's service, or running a model inside the customer's own environment. This choice is a security and compliance decision, and it belongs to the customer's security team, not only to you.
Ask four questions early. Where does the data go, and in which country? What does the provider do with it, and is it kept or used for training? Can personal details be masked or removed before sending, as Lesson 3.4 recommended? Does the customer's regulator or policy allow it? Many customers will require that the model runs inside their own cloud account, or that only masked text leaves it. Design for the rule you are given. Never paste real customer records into a public tool to test an idea.
Cost and speed
Language models are priced by how much text goes in and out, and responses take noticeable time. A pilot with a few hundred items a day hardly notices either. A system that handles a hundred thousand items a day will. Plan for it.
Several ordinary techniques keep both in check. Send only the relevant passages, not whole documents. Use a smaller, faster model for simple steps such as routing and a more capable one only where needed. Remember answers to repeated questions. Process work in batches where speed does not matter. Measure the cost per item in the pilot, so that the customer can see what a rollout would cost. Do not guess figures from memory, since prices and speeds change often, and quote the ones from the provider's current documentation when you plan.
Each rung of the ladder adds model calls, and every call adds time and cost. The simulator below shows only the time and the call count, since prices change and belong in the provider's current documentation. Set how many calls a design makes for one case, how long each takes and how many cases you handle.
At the defaults, a three-call chain with four seconds a call makes a user wait twelve seconds and uses 6,000 calls a day. An agent loop of eight calls would make the user wait thirty-two seconds and use 16,000 calls a day for the same cases.
Worked example 1: the support drafts that cited their sources
This continues the support assistant from Lesson 4.1, whose prototype once gave a customer an out-of-date refund rule. The engineer rebuilds it properly. The task is narrowed to drafting replies about statements, due dates and charges. The assistant retrieves passages from the current help pages and is told to answer only from them, to quote the passage and to say so when none fits.
Guardrails go outside the model. Any draft that mentions a refund, a legal term or an amount over a limit is sent to a senior agent, whatever the model says. The source pages carry a last-updated date, and a nightly job flags pages not updated in a year. Agents see the draft beside the source passage, and a weekly audit of twenty approved drafts checks that reviewers still catch errors.
The test set of fifty real questions gives 46 correct, with the error types in the chart. The out-of-date source error is fixed by the nightly check, the wrong facts by adding two worked examples to the prompt, and the missing warning by a guardrail. A second test of 200 cases measures 95 per cent, with a margin of about 3 points. The pilot widens, with the senior-agent route kept for refunds.
Worked example 2: reading loan documents
A lender wants the key fields, such as name, amount, tenure and date, extracted from scanned loan applications. The engineer builds a chain. A first step reads the scan and returns the fields in a fixed structure. A second step checks them with ordinary code: dates must be valid, amounts plausible and the name must match the applicant on file. Anything that fails a check, or that the model marks as unclear, goes to a person with the scan beside it.
The test set of a hundred applications shows that the model reads typed forms very well and handwritten ones much less well, with most errors in handwritten amounts. Because the check step compares the amount with the loan product's allowed range, many of those errors are caught and sent to review, which is the right place for them. The engineer reports the result by type: typed forms and handwritten forms separately, with the share sent to review. The sponsor sees that the system saves most of the typing on typed forms and does not pretend to solve handwriting.
Here is one way the result could look. Suppose seventy of the hundred applications are typed and thirty handwritten. The chart shows how well the model reads each kind and how many forms from each go to a person, which is the shape of result the engineer would report.
Worked example 3: the agent that was not needed
An operations team asks for an agent that "handles failed deliveries". The first design gives a model a set of tools, such as looking up an order, contacting the customer and rebooking, and lets it decide what to do. In testing it sometimes calls the same tool repeatedly, once rebooks a delivery to the wrong address and costs far more per case than expected.
The engineer steps back and looks at what the team really does. Almost all failed deliveries fall into four types, each with a fixed response. The better design is a router that classifies the failure into one of the four types, a fixed workflow for each, with the model used only to read the customer's message and draft the contact, and a person approving any rebooking. It is easier to test, costs a fraction as much, and its behaviour can be explained to the security team in a page.
The lesson matches the guidance quoted earlier. An agent is a powerful tool for open-ended problems, and most customer problems are not open-ended. The right test is whether a fixed workflow would do. If it would, use it.
Common mistakes
Judging by a demo. A few impressive outputs prove little. Test on real cases and measure.
Climbing the ladder too fast. Start with one prompt or one prompt with retrieval. Move up only with evidence.
Skipping grounding. For the customer's facts, supply the current documents and instruct the model to answer only from them.
Putting rules only in the prompt. Enforce important limits in code, outside the model.
A tiny test set. Fifty cases is a start. Know the margin, and extend the set before relying on the result.
Claiming a general accuracy. State the test, the number, the margin and the kinds of error.
No person in the loop. Start with draft-only, show the source and audit what reviewers approve.
Ignoring where the data goes. Settle this with the customer's security team before you send real records anywhere.
Guessing costs. Measure the cost per item in the pilot, and use current figures from the provider.
Key takeaways
In a customer setting, the stakes are real, the data is private and someone is accountable, so a demo is not evidence.
Start with the simplest design and climb the ladder of prompt, retrieval, chain, routing and agent only when tests show you must.
Five ingredients make an assistant dependable: a narrow task, grounding in the customer's documents, checkable outputs, guardrails in code and a person in control.
Build a test set of real cases with the customer, measure, read the errors by kind and rerun it after every change.
A small test gives a wide margin. State the test, the number and the uncertainty, and extend the set before relying on the result.
Settle with the customer where the data goes, plan for cost and speed at rollout volumes and keep logs of what the model produced and what people did.
Check yourself
1. Name the five rungs of the ladder of designs, from simplest to most complex.
One prompt, a prompt with retrieval of the customer's documents, a chain of steps, routing, and an agent.
2. Why enforce a rule such as "never promise a refund" in code and not only in the prompt?
A prompt is an instruction the model may not follow every time. Code outside the model can detect the case reliably and send it to a person.
3. The assistant answered 46 of 50 test cases correctly. Why is "it is 92 per cent accurate" a risky thing to tell a sponsor?
With 50 cases the true figure could plausibly be much lower or higher. State the test, the number and the uncertainty, and list the kinds of error.
4. With 200 test cases and 95 per cent measured accuracy, roughly what is the margin of error?
1.96 × sqrt(0.95 × 0.05 ÷ 200) × 100, which is about 3.0 percentage points.
5. When is an agent the right design?
When the problem is open-ended and you cannot fix the steps in advance. If a fixed workflow with a router would do, use that, because it is cheaper and easier to test.
Try this
This exercise takes about forty minutes and gives you a first evaluation. You need a language-model tool you are allowed to use and a small task with real material that is not sensitive: a club's rules, a public help page or your own notes.
1. Write the task in one line, narrow enough to test: for example, "answer questions about the club's membership rules, using only the rules document".
2. Collect or write twenty real questions, including three that the document does not answer and three that are phrased awkwardly.
3. For each, write what a good answer would be, including "the document does not say" where that is right.
4. Run the questions through the tool with your document supplied and an instruction to answer only from it and to quote the passage. Record whether each answer was correct, wrong or missing a caveat.
5. Sort the failures by kind. Write one fix for each kind.
6. Use the simulator with your number of cases and your accuracy, and write the range.
7. Write what a person should check before an answer reaches a member, and what you would automate.
A good result is a table of twenty cases with scores, a count of error kinds, a range for the accuracy and a short note on where a person should stay in the loop. This is a small version of the test that makes a customer deployment defensible.
Sources
[Building effective agents](https://www.anthropic.com/engineering/building-effective-agents), Anthropic engineering. Used for the advice to find the simplest solution possible and add complexity only when it clearly improves results, the distinction between workflows and agents, the building-block patterns of retrieval, chaining and routing, the warning that frameworks can hide prompts, and the advice to use guardrails and human checkpoints with agents. I read the full article through a summary tool.
The examples, the test results, the error counts and the simulator's defaults are the author's illustrations for teaching, not data from real deployments or evaluations. The margin of error uses a standard approximation. This lesson names no model, price or benchmark score, because those change quickly and should come from the provider's current documentation.
Tags: Forward Deployed Engineer Language models AI deployment Evaluation Customer engineering