Lesson 8.1: From notebook to production: pipelines and deployment
Learn how to turn a notebook model into a dependable service: pipelines, train-serve skew, batch vs real-time scoring, versioning, tests and safe rollout.
By the end of this lesson you will be able to explain why a model that works in a notebook can fail in use, bundle preprocessing and model into one pipeline, recognise and quantify train-serve skew, choose between batch and real-time scoring, size the servers for a scoring service, and roll a new model out safely with a shadow run, a canary and a rollback. You will watch a single unit mistake push a 5 per cent default prediction to 99.5 per cent, work out how many workers a service of 40 requests a second needs, and decide whether a 1-point jump in approval rate on a canary is real or noise. You should have read Lessons 5.1 and 6.4, since the pipeline is the working form of the "learn from training data only" rule and the windows of Lesson 6.4 decide which data a live model may see. The mathematics is multiplication, a square root and the logistic curve.
Everything so far has been about building a good model on a table in a notebook. A model that nobody uses earns nothing. The step from "this notebook gives an AUC of 0.78" to "this decision is made by the model every day, correctly, for months" is where many projects stall, and it is mostly not about statistics. It is about engineering habits: the same code in training and in use, the same inputs, a record of what ran, a way to find out when something breaks and a way to go back. This lesson covers those habits at the level a data scientist needs, whether or not an engineering team will do the heavy lifting.
Why a notebook is not a product
A notebook is a good place to explore. It is a poor place to run a business process, for reasons that are easy to list.
Notebook · ProductionCells run in any order, and state hides in memory · Code runs top to bottom, the same way every time
Data is a file you loaded once · Data arrives continuously, with gaps, late rows and new categories
The author checks the result by eye · Nobody is watching, so the code must check itself
Packages are whatever is installed · Package versions are fixed and recorded
One person, one run · Many requests, possibly at once, and a record of each
A wrong answer costs an afternoon · A wrong answer costs customers, money or a regulator's attention
None of this means notebooks are bad. It means the notebook is where the idea is proved, and the pipeline is where it is made dependable. A useful rule is that anything which will run more than once belongs in a script or a package with tests, and the notebook should call it, not contain it.
Put preprocessing and model in one pipeline
In Lesson 5.1 we saw the rule that anything computed from data, a median, a scaling constant, a category average, must be learned from the training rows only and then applied unchanged to new rows. A pipeline makes the rule automatic. It chains the preprocessing steps and the model into a single object. When you call fit, every step learns from the training data. When you call predict, every step applies what it learned, in the same order, to whatever rows it is given.
The benefit is bigger than tidiness. Without a pipeline, the preprocessing in training is one block of code and the preprocessing in production is another, usually rewritten by someone else in a different language. The two drift apart. With a pipeline, you save one object, and the service loads and calls it. There is only one definition of what "median income" or "scaled age" means, so there is nothing to disagree.
Train-serve skew
Train-serve skew is any difference between the features a model saw in training and the features it is given in use. It is among the most common and most expensive causes of a model that "worked in testing" and fails live, and it usually does not raise an error. The model produces numbers, they just are not right.
Its causes are ordinary. A feature is computed from annual income in training and from monthly income in the live system. A date is read in one format in the training file and another from the application. A column that was always filled in the history is sometimes empty live. A category appears that the training data never saw. A "last 90 days" window is defined by calendar days in one place and by the last 90 transactions in another.
Here is what a single unit mistake does. A default model has a base rate of 5 per cent, which is log-odds of ln(0.05 ÷ 0.95) = −2.94. One feature is the loan-to-income ratio, with a coefficient of 1.5. For a typical borrower in training the ratio is 0.5, because the loan is half of annual income. Now suppose the live system feeds the loan divided by monthly income, so the same borrower's ratio is 6.0. The log-odds change by 1.5 × (6.0 − 0.5) = 8.25, to −2.94 + 8.25 = 5.31, and the predicted probability of default is 1 ÷ (1 + e^−5.31) = 99.5 per cent. The model has not become wrong about the world. It has been told something false about the borrower.
Predict first: the live system feeds the model a ratio of 2.0 where training saw 0.5, a factor of four. Will the predicted default for a typical borrower rise a little, double, or rise sixfold or more? Decide before reading on.
The coefficient is 1.5 and the gap is 1.5, so the log-odds rise by 2.25 and the probability goes from 5.0 per cent to 33.3 per cent, a rise of more than sixfold. Because the logistic curve is steep in the middle, a modest-looking error in a feature is magnified into a very large error in the output. No exception is raised, the output is a perfectly valid probability, and without checks, nobody notices until the approval rate collapses.
The cures are all forms of making the two sides agree and checking that they do:
One definition. Compute features with the same code in training and in service. If the live features come from a separate feature store or database query, test that it reproduces the training values on the same rows.
A schema check. Before scoring, verify that every expected column is present, has the expected type and falls within a plausible range. A ratio above 20 or a negative age should stop the request or raise an alert, and not be scored.
A replay test. Take a few thousand historical applications, run them through the live path, and compare the live features and scores with the training values. They should match to rounding.
Monitoring of inputs. Once live, compare the distribution of each feature with its training distribution, the monitoring Lesson 8.2 covers.
Batch or real time?
The next design choice is when the model runs.
· Batch scoring · Real-time scoringWhen · On a schedule, for many records at once · At the moment of a request, for one record
Typical uses · Monthly collections lists, campaign targeting, portfolio risk reports · Loan application decisions, card fraud checks, recommendations
Latency need · Minutes to hours is fine · Often tens to hundreds of milliseconds
Features · Can use big tables and joins · Must be available fast, so often pre-computed
Complexity · Low · Higher: servers, timeouts, uptime, scaling
Failure handling · Rerun the job · A fallback rule is needed in the moment
Many problems that sound real-time are really batch. A bank that wants to target a pre-approved loan offer to customers does not need a score in 50 milliseconds. It can score everyone overnight and store the result. Choose real time only when the decision truly cannot wait, because it costs much more to build and run.
The latency budget
For real-time scoring, the whole request has a time budget, and the model is usually a small part of it. Suppose a loan application must be decided in 100 milliseconds. The network round trip takes 30 ms, fetching the customer's features from a database 40 ms, running the model 15 ms, applying the policy rules 10 ms and writing the log 5 ms. They add up to exactly 100.
The lesson of the chart is practical. When a service is too slow, the model is rarely the culprit. Features that must be calculated from raw history on every request are slow, so teams precompute them overnight or maintain them as running totals. A very large model that adds 80 ms would blow the budget, and the data scientist may need to choose a simpler model because of this, as Lesson 5.3 suggested.
How many servers?
To size a service you need a simple relation from queueing, known as Little's law: the average number of requests being handled at once equals the arrival rate multiplied by the time each takes. A service receiving 40 requests a second, each taking 100 ms, has 40 × 0.1 = 4 requests in progress at any moment. You never run a worker at 100 per cent, because any burst would then form a queue, so you divide by a target utilisation. At 60 per cent, you need 4 ÷ 0.6 = 6.7, which rounds up to 7 workers. If the rate doubles at a peak, the answer doubles.
Packaging, versions and tests
A deployed model is more than a model file. To be able to explain, reproduce and fix a decision, record the following with every release.
The model object, saved from the fitted pipeline, with a version number.
The code version, such as the commit that produced it.
The data version: which table and which date range the model was trained on, ideally a frozen snapshot, and a hash or row count that proves it.
The environment: the versions of Python and of every package, since the same model file can behave differently, or fail to load, under different library versions.
The settings: seeds, hyperparameters and the cutoff.
The results: the validation metrics and the plots, as of training.
Teams keep these in a model registry, a catalogue of model versions with their status, such as candidate, approved or retired. The registry answers the questions that arise months later: which model made this decision, and what was it trained on?
Tests catch the failures that statistics will not. Three kinds matter most.
1. Data tests check the inputs: required columns, types, ranges, and the share of missing values.
2. Code tests check the feature code on small hand-made examples, such as a borrower whose ratio you calculate by hand.
3. Model tests check behaviour: the score is between 0 and 1, a worse borrower never scores better on a feature that should be monotonic, and the model reproduces the saved validation results when run again.
A final habit is logging every decision. For each score, store the inputs, the model version, the score and the action, with a timestamp. Without it you cannot investigate a complaint, repeat a calculation for a regulator or build the monitoring of Lesson 8.2, and the outcomes that arrive later cannot be joined to the predictions that preceded them.
Rolling out safely
Even a well-tested model can surprise you in production. The aim of a careful rollout is that any surprise is small, quick to notice and quick to undo.
1. Shadow mode. The new model scores every live request, but its answers are only logged, and the old process still makes the decision. You compare its scores and its speed with the live ones at no risk.
2. Canary. The new model decides for a small share of traffic, say 5 per cent, while the rest stays on the old model. You watch the outcomes closely.
3. Gradual increase. If the canary behaves, raise the share in steps, such as 25 per cent, 50 per cent, then 100 per cent.
4. Champion and challenger. Keep running the old model, now the challenger, on some traffic or in shadow, so there is always a baseline to compare with.
5. Rollback. Keep the previous version deployable, and decide before launch what would trigger going back.
When you watch a canary, apply the sampling thinking of Lesson 2.2. Suppose the old model approves 61.0 per cent of applications and the canary, with 2,000 applications, approves 62.0 per cent. Is that a real change? The standard error of a proportion near 61 per cent with 2,000 cases is the square root of 0.61 × 0.39 ÷ 2,000 = 1.09 percentage points, so the margin at 95 per cent confidence is 1.96 × 1.09 = 2.1 points. A one-point difference is well inside the margin of noise, and the right answer is that nothing has been detected yet. A 4-point difference would be outside it and would need an explanation before the share is raised. Note that approval rate is a quick signal; the outcomes that matter, defaults, arrive months later, so a canary must also be judged on proxies that appear quickly, such as the score distribution and early delinquency.
Governance and documentation
A model that makes decisions about people or money needs a paper trail. A short model card or model document should say what the model is for and for whom, what data it used, how it was validated, its known limits, who approved it, and how it is monitored. Many organisations require independent validation and sign-off before a model goes live, and a periodic review after. Writing this early, while the details are fresh, is far easier than reconstructing it for an audit. Lesson 8.3 shows the contents in a full example.
Putting it together
A sound path from notebook to service looks like this:
1. Move the data preparation and modelling code out of the notebook into tested functions.
2. Wrap preprocessing and model in one pipeline, and fit it on a frozen snapshot.
3. Add a schema check and write the tests.
4. Save the pipeline with a version, its environment and its metrics, in a registry.
5. Choose batch or real time from the decision, and check the latency budget if real time.
6. Run in shadow, then as a canary, with a written rollback rule.
7. Log every decision, and connect the outcomes back for monitoring.
python
import json
import datetime
import joblib
import pandas as pd
from sklearn.compose import ColumnTransformer
from sklearn.impute import SimpleImputer
from sklearn.linearmodel import LogisticRegression
from sklearn.metrics import rocaucscore
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
numeric = ["loantoincome", "incomelakh", "missedpayments", "age"]
pipe = Pipeline([
("prep", ColumnTransformer([("num", Pipeline([("fill", SimpleImputer(strategy="median")),
("scale", StandardScaler())]), numeric)])),
("clf", LogisticRegression(maxiter=1000)),
])
pipe.fit(Xtrain[numeric], ytrain) # every step learns from training rows only
the contract the service will enforce: columns, plausible ranges
limits = {"loantoincome": (0, 20), "incomelakh": (0.5, 500), "missedpayments": (0, 60), "age": (18, 100)}
def validate(rows: pd.DataFrame) -> pd.DataFrame:
missing = [c for c in limits if c not in rows.columns]
if missing:
raise ValueError(f"missing columns: {missing}")
for col, (lo, hi) in limits.items():
bad = rows[col].notna() & ~rows[col].between(lo, hi)
if bad.any():
raise ValueError(f"{col}: {int(bad.sum())} values outside {lo} to {hi}")
return rows[list(limits)]
def score(rows: pd.DataFrame) -> pd.Series:
return pd.Series(pipe.predictproba(validate(rows))[:, 1], index=rows.index)
save the pipeline with what is needed to reproduce and explain it later
meta = {"version": "1.0.0", "trainedon": str(datetime.date.today()), "rows": int(len(Xtrain)),
"features": numeric, "validationauc": round(rocaucscore(yvalid, score(Xvalid)), 3)}
joblib.dump(pipe, "defaultmodel1.0.0.joblib")
json.dump(meta, open("defaultmodel1.0.0.json", "w"))
a test: the monotonic behaviour we expect, and the replay of saved results
worse, better = Xvalid.head(1).copy(), Xvalid.head(1).copy()
worse["missedpayments"], better["missedpayments"] = 4, 0
assert score(worse).iloc[0] > score(better).iloc[0]
assert abs(score(Xvalid).mean() - joblib.load("defaultmodel1.0.0.joblib").predictproba(Xvalid[numeric])[:, 1].mean()) < 1e-9
Common mistakes
Rewriting the features for production. A second implementation will differ from the first. Reuse the training code.
Fitting the preprocessing outside the pipeline. Anything learned from all the data leaks it. Put each step inside the pipeline.
No input checks. A wrong unit or a missing column produces a plausible but wrong score. Validate before scoring.
Choosing real time when batch would do. It costs far more to build and keep running.
Judging the speed by the model alone. Features and the network usually dominate the latency budget.
No version record. Without the code, data and environment, a decision cannot be reproduced or explained.
Launching to everyone at once. Use shadow mode and a canary, with a written rollback rule.
Reading a small canary difference as real. Compare it with the margin of noise before reacting.
Not logging decisions. Without logs, monitoring and investigations are impossible.
Key takeaways
A notebook proves an idea. Production needs the same code every time, checks on the inputs, fixed package versions and a record of what ran.
A pipeline bundles preprocessing and model so that one object is saved and used, and the "learn from training data only" rule is applied automatically.
Train-serve skew is a difference between the features in training and in use. A coefficient of 1.5 and a ratio fed as 6.0 instead of 0.5 turn 5 per cent into 99.5 per cent, with no error message.
Choose batch scoring unless the decision cannot wait. For real time, plan a latency budget: features and network usually cost more than the model.
Workers needed = arrival rate × time per request ÷ target utilisation. At 40 requests a second, 100 ms and 60 per cent, that is 7.
Record the model version, code, data, environment, settings and results. Test the data, the code and the model, and log every decision.
Roll out in steps: shadow, canary, gradual increase, with a rollback rule. A canary difference smaller than its margin of noise, 2.1 points in the example, has not been detected yet.
Check yourself
1. Why is it safer to save a pipeline than a bare model?
The pipeline carries its preprocessing, learned from the training rows, so training and service use one definition of every step. A bare model leaves the preprocessing to be rewritten.
2. A model has log-odds of −2 for a borrower, and a feature with coefficient 0.8 is fed 3 units too high. What are the new log-odds, and does the predicted probability go up or down?
−2 + 0.8 × 3 = 0.4. The probability rises, from 11.9 per cent to 59.9 per cent.
3. A service receives 25 requests a second, each taking 200 ms, and you want 50 per cent utilisation. How many workers?
In progress: 25 × 0.2 = 5. Workers: 5 ÷ 0.5 = 10.
4. A canary of 1,000 applications approves 55 per cent, against 52 per cent for the old model. Is the difference larger than the noise margin?
The margin is 1.96 × the square root of 0.52 × 0.48 ÷ 1,000 = 3.1 points. A 3-point difference is about equal to the margin, so it is borderline and needs more data.
5. Name two things that should be logged with every decision, and say why.
For example the inputs and the model version. They let you reproduce the score, answer a complaint, and join later outcomes to the predictions for monitoring.
Try this
Take the best model from Module 5 or 6. Set aside ninety minutes.
1. Rewrite the preprocessing and model as one pipeline, fitted on training rows only. Check that its validation score matches the scores you had before.
2. Write a validate function with a column list and a plausible range for each feature. Feed it three bad rows, one missing a column, one with a wrong unit and one with a negative age, and confirm that each is stopped.
3. Create deliberate skew: score the same rows with one feature multiplied by 12. Record how the average predicted probability changes, and use the skew simulator to check your arithmetic.
4. Save the pipeline with a version, a metrics file and the package versions. Reload it in a fresh session and check that it reproduces the same scores.
5. Write three tests: one data test, one code test on a hand-calculated row and one model test for a behaviour you expect.
6. Choose batch or real time for your decision. If real time, write a latency budget and use the workers simulator to size the service for a peak rate.
7. Write a one-page rollout plan: what runs in shadow, what share the canary gets, what you will compare, how large a difference counts as real, and the condition for going back.
A good result is a pipeline that reloads and reproduces its scores, a validation function that stops bad input, and a rollout plan someone else could follow. The next lesson looks at what happens after launch: how to notice that the world has moved and what to do about it.
Tags: Data Science Model deployment ML pipelines Train-serve skew MLOps