Lesson 8.1: Case study: deploying an AI assistant at a lender
Follow one forward deployed engineer through fourteen weeks of a lender's AI assistant, from vague request to measured result, and see the lessons in use.
By the end of this lesson you will have followed a complete forward deployed project from the first conversation to a renewal decision, and be able to say what was decided at each stage, what the alternatives were and what the engineer would do differently. You will see how the lessons of the earlier modules fit together on one real-style project, work through the decisions yourself before reading the engineer's choice and use a simulator to rebuild the project's economics. This is the first of two case studies that open the last module. It is a composite written for teaching: the lender, the people and every number are invented, and are meant to be realistic rather than reported. Read it as a worked example, not as a history of a real company. You should have read at least the first three modules, since the case refers to them often.
Reading about a skill is not the same as watching it used under pressure. In a real project, nothing arrives labelled. The request is vague, the data is untidy, the sponsor is optimistic and the calendar is short, and the lessons you learned one at a time have to be applied together, with judgement about which matters now. A case study is the closest thing to that on paper. As you read, pause at each decision point and decide what you would do before you see what the engineer did. The value is in the gap between the two.
The setting
Vikas Finance, which is an invented name, is a mid-sized non-banking lender in India with about 300,000 active borrowers and a support team of sixty agents who handle, among other things, questions about statements, due dates and charges. A software company, which we will call the vendor, has sold Vikas a platform that includes a language-model assistant. The vendor's forward deployed engineer, whom we will call Asha, has been assigned to make it useful.
The sale was made on a demo. The head of support, Rohan Mehta, wants "an assistant that handles our customer questions and reduces our call volume by 30 per cent". The vendor's account executive has told him it is achievable in three months. Asha reads the contract the day she is assigned, and finds that it names a three-month pilot, a target of "significant reduction in support effort" and an extract of data to be supplied "in the first month". It says nothing about accuracy, languages or who reviews what. This is the kind of starting point most projects have.
Week 1: the request is not the problem
Predict first: Rohan asks Asha to start building the assistant. What do you do in the first week? Decide before you read on.
Asha does not start building. She applies Lesson 2.1: find the problem behind the request. She asks for the last two hundred support contacts and the reasons the agents recorded, and listens to twenty calls. The picture, in rounded numbers, is that about 30,000 contacts arrive a month, and that three reasons account for most of them: questions about charges on the statement, about the due date and about how to make a payment. The charge questions are the largest, around a third, and many come from borrowers who cannot read the internal codes on the statement, a finding that matches the example in Lesson 2.1.
She writes the problem in one sentence: borrowers contact support about statement charges because the charges appear under internal codes, costing about 10,000 contacts a month and about four minutes of agent time on each. She shows it to Rohan, who says, "I knew it was charges, but I did not know it was this many." The target of 30 per cent, she notes, would mean about 9,000 fewer contacts, which is close to everything she has just counted, so it would be met only if nearly all charge questions disappeared. She tells him this plainly, as Lesson 6.2 advises, and proposes a different framing: draft replies for the agents to review, and measure time per case and customer satisfaction, rather than promise a call reduction.
Rohan is not delighted. He had promised his own manager the call figure. But he agrees to test the drafting approach first, on the condition that the call data be tracked too.
Week 2: people and data
Asha builds the stakeholder map from Lesson 2.2. The sponsor is Rohan. The champion is a senior agent, Divya, who has wanted better tools for years. The users are sixty agents, some of whom suspect the assistant is a step toward replacing them, which Asha records as a risk, privately. The gatekeepers are the central IT team, which owns the knowledge base and the case system, and the compliance officer, who must approve any wording that reaches a customer. The approver for budget is the finance controller.
She audits the data, following Lesson 3.1. The sources are the knowledge base of help pages, the case system with past tickets and replies, and the statement templates. She asks for a thousand past tickets with their replies. The profile shows two problems. First, the help pages were last updated at different times, and two of them disagree about the late fee. Second, about 6 per cent of tickets have no reason code. She notes both, asks the steward who owns the help pages to reconcile the late-fee pages and starts the access requests for the case system immediately, since the data team estimates three weeks. She also starts the compliance review of reply templates, which takes two weeks, so that it will finish while she builds.
Weeks 3 and 4: scope and success
Decision point. The sponsor wants the assistant to answer all customer questions, in several languages, through chat and email. What do you scope?
Asha uses Lesson 2.3. She proposes a pilot, not a project. The question: does a drafted reply, grounded in the help pages and reviewed by an agent, reduce the time to resolve a statement-charge question without lowering customer satisfaction? The boundary: one team of twenty agents, questions about statement charges and due dates only, English and Hindi, email only, eight weeks of running with two weeks of review. Out of scope: payments, complaints, legal questions and refunds, which go straight to a person.
The metric: the median time from a ticket's arrival to the first correct reply. Baseline from the past tickets: 42 minutes. Target: 25. Guardrails: customer satisfaction at least as high as the baseline of 4.3 out of 5, and no reply with a factual error found in a weekly audit of twenty replies. Comparison: a control team of twenty agents who do not use the assistant in the same weeks.
She writes the one-page plan from Lesson 2.4, with ten parts, a timeline that puts the slow approvals first and a risk list that includes the disagreeing help pages and the agents' fears. Rohan reads it, edits two lines and approves it. The finance controller asks for the cost, and Asha gives a figure with a range, and an honest statement of what the pilot will and will not prove.
Weeks 5 to 7: build, buy, harden
Decision point. What do you build, and what do you buy or reuse?
Using Lesson 4.2, Asha splits the solution. She buys the language model through the vendor's existing platform, since the customer's compliance team has already approved it and runs it inside the vendor's approved environment, with no customer data stored by the provider beyond the request. She reuses the lender's case system for storing drafts and replies. She configures the reporting tool the lender already owns for the dashboard. She builds only what is specific: the retrieval of the right help-page passages, the prompt that tells the model to answer only from them and to quote the source, the guardrails and the review screen.
Following Lesson 4.3, she starts simple: one prompt with retrieval, no agent. The guardrails live in code: any draft that mentions a refund, a legal term or an amount over a limit goes to a senior agent, whatever the model says. She builds a test set with Divya: fifty real questions with good answers written by the senior agents, including five the help pages do not answer. The first run is right on 38 of 50. Reading the failures, she finds that eight are caused by the two disagreeing late-fee pages and four by a retrieval problem that returns the wrong passage when a question has two parts. The late-fee pages are fixed by the steward. The retrieval is changed to search in two steps. The second run: 46 of 50, with the four errors split between a wrong fact, an old source and two missing warnings. She adds a guardrail for the warnings and a nightly check that flags help pages not updated in a year.
She reports the result as Lesson 4.3 recommends, with its limits: on fifty cases, the plausible range is wide, so she extends the test to two hundred before the pilot starts. The result is 95 per cent, with a margin of about three points.
She also follows the prototype rule of Lesson 4.1. The first version, on day four, is labelled a prototype and shown to Divya and two agents. They give three corrections, among them that drafts must show the source passage beside them. She keeps a shortcuts list, including "no retry if the model service fails" and "reads help pages from a copy", and agrees a two-week hardening phase before any agent uses it on a real ticket.
The test results tell the story of the build. The chart shows the share of cases answered correctly at each stage.
Week 8: the demo that landed
Decision point. How do you show it to the people who matter?
Asha runs demos as Lesson 4.4 suggests, and in this week the important one is for the twenty agents who will use the assistant, and their team leader. Using Lesson 6.1, she builds the story around an agent's morning. "Divya has forty emails in her queue. Eleven are about statement charges." She shows one flow: an email arrives, a draft appears with the quoted help-page passage beside it, Divya reads, edits one line and sends. She shows what does not work: two kinds of question the assistant refuses and sends to a senior agent. She asks for reactions.
The agents' questions are about themselves: "Will this be used to count how many replies I send?" and "What if it writes something wrong and the customer complains?" Asha asks Rohan to answer the first in the room, and he says the tool will not be used to rank agents, and he puts it in writing the same day. For the second, she shows the review step and the audit. Two agents volunteer as first users. Training, following Lesson 6.4, is thirty minutes on three tasks, on real tickets, with Divya as champion and an office hour each morning in the first week.
Weeks 9 to 12: running, and a bad week
The pilot runs for four weeks within the eight. Asha releases in the way Lesson 5.2 describes: one package, version numbers and a rollback plan, to the pilot team in two stages, first six agents and then twenty. The dashboard tracks the four signals of Lesson 5.3 and freshness of the help pages, and she has alerts on silence: if no draft is produced during business hours for thirty minutes, she is called. A runbook describes the first checks.
In week ten there is a bad day. The vendor's model service has a slow period, and drafts take two minutes instead of ten seconds. Agents start to write replies by hand. The alert fires, the runbook says to check the provider's status page and to switch the assistant to "drafts on request" so that agents are not waiting. Asha does so within fifteen minutes and tells Rohan in two lines. By afternoon the service recovers. The agents' reaction surprises her: a few say that the fallback made them trust it more, because the tool told them plainly what was happening.
Midway, she gives the weekly update in the seven-part format. In week eleven it is amber: the satisfaction guardrail is holding, but time to first reply has dropped only from 42 to 31 minutes against a target of 25, and two agents are ignoring the drafts. She says so in the first two lines, with the decision she needs: whether to extend the pilot by two weeks and retrain. Rohan agrees, partly because the earlier amber updates taught him to trust her colours.
She applies Lesson 6.4 to the two who ignore it. She sits beside them. One finds the review screen slow, which Asha fixes. The other says she does not trust drafts about charges, because the first one she read contained a wrong fee. Asha shows her the audit result and the fix to the source page, and asks her to flag any wrong draft, which she does, three times in the next week. By week fourteen, she is a user.
Week 14: the result, and the value
The pilot reports its result, with the control team. Time to the first reply: 26 minutes for the pilot team against 41 for the control, a fall of 15 minutes. The target was 25, so it is nearly met. Satisfaction: 4.4 against 4.3, within the guardrail. The weekly audit found one factual error in the twenty-reply samples across all eight weeks, caught by the agent before sending.
Asha uses Lesson 7.3 to turn it into value. With the finance controller, she establishes the figures. The control comparison gives the 15 minutes of reduction. Agents are salaried, so the cashable saving is nil in the short term, and she says so. The saving is capacity. The pilot team's twenty agents handle about 4,900 cases of these question types a month, and their logged handling time per case fell from about 6 minutes to 4.4, a saving of 1.6 minutes, which is about 130 hours a month. At the cost per agent hour that finance supplies, ₹300, that is about ₹39,000 a month, which she labels as capacity, not cash. For the whole support team of sixty and the same question types, it would be roughly three times that. She also reports the call reduction, which was modest, about 4 per cent, and does not claim more. She notes that the real prize, the one-third of contacts about statement charges, is better addressed by changing the statement, which is outside her pilot, and recommends it.
Rohan takes a one-page value report to his manager. The honest figures, with the capacity framing, are less dramatic than the 30 per cent he had been promised, and they hold up. The finance controller approves a rollout to all sixty agents for the same question types.
The result against the target is easiest to read as four bars: the baseline, the target Asha agreed, the control team and the pilot team.
Handover, productising and what followed
Asha plans the handover from week one, as Lesson 5.4 says. She trains Divya and a second agent, as the owners of day-to-day operation, and hands the runbooks to the central IT team. The four stages run during the pilot, and in week fourteen Divya and the second agent perform a release and handle a rehearsed incident without her. She sets the support model: agents' questions to Divya, operations to the IT team, defects to the vendor. Her own time falls from thirty hours a week to two over about two months.
She also feeds the product team, using Lesson 7.2. Her field brief covers three things the pilot showed: a source-quote display that every deployment needs, a nightly check for stale help pages and a review step with senior routing. The pattern log shows two other customers with the same needs. The first two are built into the product in the next quarter, and the third becomes a configuration. The statement redesign is taken up by Vikas's own team, with a note from Asha showing the volume of charge questions by code.
What went well, and what she would change
It is worth listing the choices that mattered most.
Finding the problem before building. The week-one analysis changed the whole project, from a call-reduction promise to a drafting pilot with an honest metric.
Starting the slow approvals early. Access and compliance review ran in parallel with the build.
A real test set. It exposed the disagreeing help pages, which no amount of prompt-writing would have fixed.
Guardrails in code. The refund and legal rules did not depend on the model following a sentence.
Honest colours and early bad news. The amber in week eleven cost nothing, because trust had been built.
Reporting capacity as capacity. The value held up under finance's questions.
And what she would change.
Talk to the agents earlier about their fears. Their worry about ranking surfaced in week eight, and she would have raised it in week two.
Involve the statement owner in week one. The biggest finding was outside her scope, and she learned it late. An earlier conversation could have started the statement change sooner.
Train a second champion in each shift. The night shift had no champion, and its adoption lagged.
Rebuilding the economics
You can reconstruct the project's economics from its main numbers. The assistant drafts a reply, and an agent reviews it. Some cases are routed to a senior agent, which costs extra time. The saving per case is the time of handling it by hand, minus the time with a draft and a review, including the extra time on escalated cases.
The simulator below does the sum. Set the number of cases a month, the minutes to handle one by hand, the minutes with a draft and review, the share escalated to a senior agent and the extra minutes an escalation takes, and the cost of an hour of agent time. It shows the minutes saved per case, the hours and the money saved each month, and plots the saving per case as the escalated share grows.
The result is capacity, as Asha insisted, and not cash, unless hours or hiring change. The numbers are the case's invented ones. Use your own to see what a similar assistant would offer in your setting.
Try three experiments. First, keep the defaults and read the monthly saving. Second, raise the escalated share from 15 to 40 per cent, which describes a task with more risky cases, and see the saving fall. Third, raise the review time to match the manual time, which describes drafts that agents do not trust and rewrite, and see the saving vanish.
At the defaults, handling a case by hand takes 6 minutes, and with a draft, review and 15 per cent escalation it averages 3.5 plus 0.9, which is 4.4. The saving is 1.6 minutes a case, the figure the pilot measured. Applied to all 30,000 contacts a month, which the defaults do for illustration, that would be 800 hours, worth about ₹2.4 lakh in capacity. In practice the pilot measured the saving only on the question types it covered, a fraction of all contacts, and other types may save less, which is why Asha recommended rolling out to the types she had measured and not to everything.
Questions to ask of any case
When you read a case, or live one, a short list of questions turns it into learning.
1. What was the request, and what was the real problem? How did the engineer find out?
2. Who were the people that decided, and who could have blocked it? When did the engineer meet each?
3. What did the engineer choose not to build, and why?
4. Where was the evidence, and how was it checked?
5. What went wrong, and how was it communicated?
6. How was the result measured, and what was claimed?
7. What happened after: handover, adoption, product?
8. What would you have done differently, and at which week?
Common mistakes
Accepting the target as given. Asha's first act was to test whether the 30 per cent was achievable.
Building before the test set. Without real cases with known answers, nobody knows whether the assistant works.
Leaving fears unaddressed. Users who fear ranking will resist, however good the tool.
Claiming cash for capacity. The value report must say which it is.
Stopping at the pilot. The project ends with a handover and a field brief, not with a metric.
Key takeaways
A project that goes well is mostly the same ordinary practices, applied in sequence and in time: find the real problem, map the people, audit the data, scope a pilot, build the least that works, test on real cases, demo early, communicate honestly, measure value and hand over.
The most valuable moves in the case were early: the week-one problem analysis, the early approvals and the real test set.
Honest early updates, including an amber colour, build the trust that lets a project survive a setback.
The value was reported as capacity, with the method shown, and it survived finance's questions because it was modest.
The end of the project is a handover, a field brief and a recommendation, not a report of the metric.
Check yourself
1. What did Asha do in week one, and why did it change the project?
She analysed the real support contacts to find the problem behind the request. It showed that the 30 per cent target was close to everything she had counted, so she proposed a drafting pilot with an honest metric.
2. Why did she build a test set before the pilot, and what did it reveal?
To measure the assistant on real cases. It revealed that disagreeing help pages were causing many of the errors, which no prompt change would have fixed.
3. Why was the value reported as capacity and not as cash saved?
Agents are salaried, so the time freed does not reduce cost unless hours or hiring change. Saying so made the figure survive finance's questions.
4. With 20,000 cases, 5 minutes by hand, 3 minutes with a draft and 10 per cent escalated at 5 extra minutes, what is the saving per case and in hours a month?
Assisted time is 3 + 0.5 = 3.5 minutes, so the saving is 1.5 minutes a case. 20,000 × 1.5 ÷ 60 = 500 hours.
5. Name two things Asha would change.
Talk to the agents about their fears earlier, involve the statement owner in week one, or train a second champion for the night shift.
Try this
This exercise takes about forty minutes and turns the case into your own review. Use this case, or a real project from your work.
1. Write the case in ten lines: the request, the real problem, the people, the pilot, the metric, the result and the value.
2. For each decision point in the case, write what you would have done before reading what Asha did, and compare.
3. Pick the three choices you think mattered most, and write why.
4. Pick one thing you would do differently, and write the week you would do it and what you would say.
5. Use the simulator with your own numbers for a similar assistant in a setting you know, and write what it says.
6. Write a one-page field brief on one thing the case showed that a product team should build.
7. Write the first two lines of the weekly update Asha would send in week eleven.
A good result is a short case summary, an honest comparison with your own instincts, one thing you would change and a field brief. Reviewing a project this way, after each one, is the fastest way to improve.
Sources
This is a composite case written for teaching. The lender, the people, the project and every figure are invented and meant to be realistic. It draws on the methods of the earlier lessons in this track and on the sources cited in them, and none of it reports a real company's project. The simulator's defaults are the case's invented figures.
Tags: Forward Deployed Engineer Case study AI assistant Lending Customer deployment