Lesson 2.3: Scoping a pilot with a success metric
A good pilot is a small experiment with a clear metric, a baseline and an exit rule. Learn to scope one, pick the number that proves it, and size it.
By the end of this lesson you will be able to turn a problem and a map of people into a scoped pilot: a small, time-boxed experiment with a clear boundary, one success metric, a baseline, a target and a rule for what happens next. You will learn five ways to draw the boundary of a pilot, five properties of a good metric and a plain way to check that your pilot is big enough to prove anything. You will see three pilots scoped, with numbers, and use a simulator to find out how many accounts a pilot needs before a result can be trusted. This lesson follows Lessons 2.1 and 2.2, where you found the problem and the people. You should be comfortable with percentages, and nothing more technical is needed.
Many projects with customers fail in a quiet, avoidable way. They begin with a vague aim such as "improve collections" or "make support better", they run for months and at the end nobody can say whether they worked. The software may be fine. The trouble is that the project never agreed what success would look like, so every result can be argued about. A pilot is the cure: a small piece of the work, arranged so that it produces a clear answer. Scoping is the craft of arranging it.
What a pilot is, and what it is not
A pilot is a time-limited trial of a solution on a limited slice of the real work, designed to answer a question: does this work well enough, here, to justify doing more? It differs from three things people confuse it with.
A demo shows that something could work, usually on clean sample data, and proves little about the customer's real world.
A proof of concept tests whether something is technically possible, often in a sandbox. A pilot tests whether it produces value in real use.
A project is the full delivery, with a long plan and a large budget. A pilot is the small step that decides whether to commit to one.
The important word is question. A pilot without a question is a small project that has not yet found out whether it was worth starting. Before you plan anything, write the question down in one line: "Does sending a reminder in the borrower's own language raise the share of first-month overdue accounts that pay within seven days?" That sentence already hints at the scope, the metric and the comparison.
A pilot also has an end. It stops after a set time and produces a decision: expand, change, or stop. Ending is the feature, not a failure. A customer who knows a trial will finish, and what it will decide, is easier to bring along than one who is asked for an open-ended commitment.
Five ways to draw the boundary
Scope is a boundary, and there are five places to draw it. Pilots that go wrong usually left one of the five open.
1. Who. Which users or customers are included? One team, one branch, one product line. Smaller is easier to manage and to compare, but must be big enough to measure (more on this below).
2. What. Which part of the problem is covered? One use case, one type of request, one segment of accounts. A pilot that tries to cover everything has no boundary at all.
3. When. How long does it run and when does it start? Four to twelve weeks is common. Include time for setup and for measuring, not only for running.
4. Where. Which systems, data and environment? Real data from one system, in the customer's own cloud account, is typical. Say what is excluded.
5. How much. What is the budget, effort and support? Name the people on each side and the hours they can give.
Write each answer in a sentence, and next to it write what is deliberately left out. The "out of scope" list is as important as the "in scope" list, because it is what you point to when someone asks for one more feature in week five.
Dimension · In scope · Out of scopeWho · Collections team A, 12 callers, one city · Other cities, field agents
What · Reminders for first-month overdue accounts in the smallest overdue band · Later bands, legal notices, settlement offers
When · Four weeks of setup, six weeks of running, two weeks of review · Anything after week twelve
Where · Nightly file from the loan system, delivered to the lender's own cloud account · Real-time integration, the mobile app
How much · Two engineers from our side, one analyst from theirs, four hours a week of the head of collections · A dedicated customer team
## Choosing the success metric
The metric is the number that will tell you whether the pilot worked. It is the most important choice in the scoping, and it should be agreed with the customer before anything is built. A good metric has five properties.
1. It matters to the customer. It should connect to something the sponsor already cares about: money, time, risk, quality. If the sponsor does not care about the number, a good result will not move them.
2. It is measurable now. You can calculate it from data that exists or can be collected within the pilot. If you cannot measure today's value, you cannot show a change.
3. It responds within the pilot. A metric that takes a year to move, such as annual customer loss, cannot judge a ten-week pilot. Choose one that shows an effect in weeks, and link it to the long-term outcome in words.
4. It is owned by the customer. The customer should be able to calculate and defend it without you. A metric that only you can compute will be doubted.
5. It is hard to game. If someone can improve the number without improving the outcome, expect them to. Pair it with a guardrail.
Predict first: a customer asks you to run a pilot that will "improve collections". Think of one number that would show whether the pilot worked, and decide whether you prefer it to "total money recovered this quarter". Decide before you read on.
A strong choice is something like "the share of accounts in the first overdue band that pay within seven days of the reminder". It matters to the sponsor, you can calculate it from the nightly file, it responds within a week and callers cannot easily game it. "Total money recovered this quarter" is the thing the sponsor really wants, but it is a poor pilot metric: it moves slowly, depends on many things you do not control, such as the economy and the mix of accounts, and could rise or fall for reasons unrelated to your pilot. A good pilot metric is a leading indicator, a number that moves early and is tied to the lagging outcome the customer cares about.
The chart below makes the point about speed. For a pilot of about ten weeks, only the metrics that move within weeks can judge it.
Baseline, target and guardrail
A metric on its own is not enough. Three more things turn it into a test.
The baseline. What the number is today, measured the same way. Measure it from real data over a stretch long enough to smooth out normal swings, for example the last three months. Without a baseline, any result can be called good or bad.
The target. What number would count as success, agreed in advance. A useful way to set it is to ask the sponsor: "What is the smallest improvement that would make you want to do this for the whole team?" That is the target. It is better to agree a modest, believable target than a heroic one that sets the pilot up to fail.
The guardrail. A second metric that must not get worse while the first improves. If the pilot raises the share of accounts that pay on time but also raises complaints or messages sent after hours, the first number is not telling the whole truth. Choose one or two guardrails and write down the limits.
For the collections pilot: the baseline is 41 per cent of first-band accounts paying within seven days. The target is 48 per cent, the sponsor's "worth rolling out" threshold. The guardrail is that complaints about contact stay at or below today's rate of about 2 per 1,000 contacts, and that no message is sent outside the hours the regulator allows.
Comparing with a control
In the chart above, the pilot group reached 46 per cent, short of the 48 per cent target. What the chart also shows is a control group, a similar set of accounts that did not get the new reminders during the same weeks. It stayed at 41 per cent. Without the control, you could not tell whether the improvement came from your reminders, from the end of a festival season, a new law or a change in the economy. With it, you can say that the pilot group did five points better than a comparable group over the same period.
There are three common ways to compare:
A control group. Split similar accounts into two groups, at random if you can, and treat only one of them. This is the strongest option.
Before and after. Compare the same group before and after. It is easy, but anything else that changed in the period can mislead you.
A matched comparison. If random splitting is impossible, pick a comparison group that resembles the pilot group, such as another branch, and check that the two looked alike before the pilot began.
Whichever you choose, write it into the plan. Customers who see the method agreed before the results are less likely to dispute the result.
Is the pilot big enough?
There is one more check, and teams often skip it. A result is only trustworthy if the pilot is large enough. If you try a reminder on forty accounts and ten more pay than in the control, the difference could easily be luck. The more accounts you include, the smaller the luck factor.
A rough rule uses the spread of the result. When you compare two groups of the same size, the typical random wobble in the difference between them, in percentage points, is about the square root of two times the rate times one minus the rate, divided by the group size, all times one hundred. A difference smaller than about twice that wobble is hard to distinguish from luck. This is a simplification, but it is a useful warning light.
The simulator below does the arithmetic. Enter the baseline rate, the number of accounts in each group and the improvement you saw. It shows the typical wobble, the size of difference you would need to be reasonably sure and how many wobbles your result is. A result of less than about two wobbles should be called "promising" and not "proven".
Try three experiments. First, keep the defaults and see that a five-point improvement with 500 accounts per group is not yet convincing. Second, raise the group size to 2,000 and see the same improvement become clear. Third, lower the baseline rate to 10 per cent, and notice how the wobble changes for rare events.
At the defaults the wobble is about 3.1 points, so an improvement needs to be about 6.1 points to be reasonably sure, and a 5-point improvement counts for only 1.6 wobbles. This is the common situation: a pilot sized for convenience, not for evidence, produces results that everybody can argue about. If you can pick the size, work it out before you start and write it into the plan. If you cannot, say so honestly and call the result a signal, not a proof.
A pilot is big enough only if it runs long enough to collect the accounts it needs. The simulator below turns the group size you need into weeks of running.
At the defaults, 200 eligible accounts a week split evenly give 100 a week to each group, so 600 accounts take six weeks, which is the pilot length in the example below.
Worked example 1: the collections reminder pilot
The question: does a reminder in the borrower's language raise the share of first-month overdue accounts that pay within seven days?
Scope: the first overdue band, in one city, for six weeks of running, through the nightly file, with a random half of eligible accounts as the control. Metric: the share paying within seven days. Baseline: 41 per cent. Target: 48 per cent. Guardrails: complaints no higher than 2 per 1,000 contacts and messages only within permitted hours.
Size: there are about 1,200 eligible accounts over six weeks, so about 600 per group. The simulator gives a wobble of about 2.8 points at 41 per cent, so a difference of about 5.6 points is needed to be reasonably sure. The target of 7 points over the baseline is therefore measurable, but a result of 4 or 5 points would be inconclusive. The engineer tells the sponsor this before the start: "If we see seven points or more, we can say it worked. If we see between three and five, we will need to run it longer."
This honesty is useful. It stops the customer from reading a modest result as proof, and it prepares them for the case where the answer is "not yet".
Worked example 2: the bed screen pilot
The question: does a single screen showing free beds cut the time bed managers spend finding a bed for a patient?
Scope: two of six hospitals, emergency admissions only, eight weeks. Metric: the median minutes from a bed request to a confirmed bed. Baseline: 18 minutes, measured from the request log over the previous two months. Target: 8 minutes. Guardrail: the screen's bed counts must agree with a manual check in at least 95 per cent of spot checks, so that speed does not come from wrong information.
The control is the four hospitals that do not yet have the screen. Because the hospitals differ in size, the engineer checks that the two groups had similar request times before the pilot. The result after eight weeks, in this illustration, is a median of 9 minutes in the pilot hospitals against 17 in the others, with 96 per cent accuracy. The target is nearly met, the guardrail holds and the sponsor has what is needed to extend the screen.
The key choice here was the guardrail. A screen that shows a bed as free when it is not would make the first metric look excellent and harm patients. The guardrail kept the pilot honest.
Worked example 3: the support assistant pilot
The question: does an assistant that drafts replies for support agents shorten the time to a first reply without lowering customer satisfaction?
Scope: one product line, twenty agents, six weeks, drafts only, with every reply read and sent by an agent. Metric: median time to first reply. Baseline: 42 minutes. Target: 25 minutes. Guardrails: customer satisfaction at least as high as the baseline of 4.3 out of 5, and no reply sent with a factual error found in an agent's weekly audit of a sample of twenty replies.
The plan includes a clear boundary: the assistant does not answer refund or legal questions, which go straight to a person. This sentence, written into the scope, prevents a hard conversation in week four when someone asks for refunds to be included. When the request arrives, the engineer points to the out-of-scope list and offers to add it to the next phase if the pilot works.
Exit rules: what happens next
The last piece of scope is the decision at the end. Agree it before you start, and write it as a short table of "if this, then that".
If the result is · Then weAt or above the target, guardrails held · Plan the full rollout, with a new scope and budget
Between the baseline and the target · Run a second pilot with the changes we learned, or extend the first
At or below the baseline, or a guardrail broken · Stop, write up what we learned and decide whether the problem or the approach was wrong
Agreeing this early removes the argument at the end. It also reassures a nervous customer: a pilot that can stop costs less than an open-ended commitment. The same rule applies on your side. If the result is poor, you should be ready to say so, and to be useful in finding out why.
Holding the boundary
Once the pilot is running, requests to widen it will arrive. A new manager wants another team added. Someone wants one more report. The sponsor mentions an idea over lunch. Most of these are reasonable and many are exciting, and a good engineer wants to say yes. The trouble is that each addition makes the pilot harder to measure and later to finish.
A simple practice helps. Keep a "parking lot" list, visible to the customer, of every request that is outside the boundary. When one comes in, write it down, say that it sounds useful and say that it will be considered after the pilot's decision. The list does two jobs. It shows the customer that their ideas are heard, and it becomes the starting point for the next phase if the pilot succeeds. Some requests will be important enough to change the pilot. In that case, change the scope formally, in writing, with the sponsor, and say what moves in exchange: time, budget or another item.
Common mistakes
No question. A pilot without a question cannot be passed or failed. Write the question in one line.
A vague metric. "Improve collections" is an aim. Choose a number, with a definition anyone can recompute.
No baseline. Without today's figure, any result can be spun. Measure it from real data first.
A heroic target. A target nobody believes is a plan to disappoint. Ask for the smallest result worth rolling out.
Forgetting the guardrail. A number that improves while the outcome worsens is a warning. Pair every metric with something it must not harm.
A pilot too small to prove anything. Check the size before you start, and call small results signals, not proofs.
No comparison. Without a control, outside changes can explain your result. Use a control or matched group.
Letting scope creep. Use a parking lot and change the scope only in writing.
No exit rule. Agree what each result will lead to before you see it.
Key takeaways
A pilot is a small, time-boxed experiment that answers a question, ending in a decision to expand, change or stop.
Scope is a boundary drawn on five dimensions: who, what, when, where and how much, with a list of what is deliberately left out.
A good metric matters to the customer, can be measured now, responds within the pilot, is owned by the customer and is hard to game.
A metric needs a baseline, a target and one or two guardrails. Set the target as the smallest result worth rolling out.
A control group or matched comparison separates your effect from everything else that changed.
Check that the pilot is large enough to prove anything, and agree the exit rules before the results arrive.
Check yourself
1. Name the five dimensions of scope.
Who, what, when, where and how much.
2. Why is "total money recovered this quarter" a poor pilot metric, even though the sponsor cares about it?
It moves slowly and depends on many things the pilot does not control. A leading metric such as the share paying within seven days responds within weeks.
3. What is a guardrail, and give an example.
A second metric that must not get worse while the first improves, such as complaints per 1,000 contacts staying at or below today's rate.
4. A pilot has 800 accounts per group, a baseline of 30 per cent and shows a 4-point improvement. Using the simulator's rule, is it above about two wobbles?
The wobble is sqrt(2 × 0.30 × 0.70 ÷ 800) × 100, about 2.3 points. Four points is about 1.8 wobbles, so it is promising but not proven.
5. Why agree the exit rules before the pilot starts?
It removes the argument at the end, prepares the customer for a modest result and keeps the pilot from drifting into an open-ended commitment.
Try this
This exercise takes about forty minutes and produces the scope section of a pilot plan.
1. Choose a problem from Lesson 2.1's exercise, or any repeated problem you know well. Write the pilot question in one line.
2. Draw the boundary: write one sentence each for who, what, when, where and how much. Next to each, write one thing that is left out.
3. Choose a metric. Check it against the five properties and write one line for each.
4. Write the baseline and say where it would come from. If you cannot find it, write who you would ask. Then write the target and the reason for it.
5. Write one guardrail and its limit.
6. Choose a way to compare: control group, before and after, or a matched group. Use the simulator to check whether your likely pilot size can show the target improvement.
7. Fill in the three-row exit table.
A good result is a one-page scope with a question, a boundary, a metric with a baseline and a target, a guardrail, a comparison method and an exit table. It is the middle section of the one-page plan you will write in the next lesson.
Sources
The pilots, figures, thresholds and the rule of thumb in the simulator are the author's illustrations for teaching. The rule of thumb is a standard approximation for comparing two proportions and is deliberately simplified. It is not data from a real company or study.
Tags: Forward Deployed Engineer Pilot design Metrics Scoping Beginner