Lesson 8.2: Monitoring drift and retraining
Learn how to detect that a live model has gone stale: input and score drift with PSI, bad-rate control limits, Gini decay, recalibration and when to retrain.
By the end of this lesson you will be able to explain why a model that was right at launch gets worse, tell apart the main kinds of drift, build a monitoring plan that works even when outcomes arrive months late, judge whether a bad-rate or Gini movement is signal or noise, shift a model's calibration with a single number, and decide between recalibrating, refitting and rebuilding. You will follow a population stability index as it climbs from 0.002 to 0.121 over six months, put a control band of 4.1 to 6.6 per cent around an expected bad rate of 5.3 per cent, and convert a gap between predicted and observed defaults into a shift of 5.6 points on a scorecard. You should have read Lessons 6.3 and 6.4, since the population stability index, calibration and windows are the working parts, and Lesson 8.1, which explains how decisions are logged. The mathematics is proportions, a square root, a logarithm and the margin of error from Lesson 2.3.
A deployed model is a photograph of the past. It learned the relationship between inputs and outcomes from a period of history, and every day the world moves a little further from that period. Customers change, products and policies change, the economy turns, fraudsters adapt and the data pipeline is quietly altered. A model that gave an AUC of 0.78 at launch may give 0.70 a year later and nobody is told, because a model does not fail loudly. It simply goes on producing numbers. Monitoring is the discipline of noticing, early and cheaply, that those numbers no longer deserve trust, and retraining is the discipline of restoring them.
What goes wrong: kinds of drift
The word drift covers several different things, and the cure depends on which one you have.
Kind · What changes · Example · Typical signData (covariate) drift · The mix of inputs · A new partner channel brings lower-income applicants · Feature and score distributions move
Concept drift · The link between inputs and outcome · The same income now carries more risk in a downturn · Bad rate differs from prediction at the same score
Prior (label) drift · The overall rate of the outcome · Defaults rise across the board · Average bad rate rises everywhere
Data quality change · The meaning or availability of a field · A source system starts sending monthly income · Sudden jumps, missing values, new categories
Policy or process change · Who is approved or how · The cutoff is raised · Population and outcomes shift together
Two of these deserve a warning. First, data drift is easy to see and does not always matter: if the mix of applicants changes but each type behaves as before, the model remains correct for each of them. Concept drift is the dangerous kind, because the model is confidently wrong, and it can only be seen from outcomes. Second, the cheapest explanation is the data quality change. Before assuming that the world has changed, check that the pipeline of Lesson 8.1 still sends what the model expects.
Three layers of monitoring
A good plan watches three layers, because each answers a different question and arrives at a different speed.
Layer · Question · Examples · Needs outcomes? · SpeedOperations · Is the service healthy? · Errors, timeouts, latency, share of missing inputs · No · Minutes
Inputs and scores · Is the population the model sees still like the one it learned from? · Population stability index of the score, characteristic stability of each feature, new categories · No · Days
Outcomes · Is the model still right? · Bad rate against prediction by band, Gini, KS, calibration · Yes · Months
The middle layer is the early warning system. In lending, the true outcome, default within twelve months, takes a year to observe, so a model could drift for a year before the performance numbers show anything. Input and score monitoring needs no outcomes and can be run every week. The outcome layer is the final judge and arrives late, so you supplement it with early indicators: a proxy for the final outcome that shows up sooner, such as the share of accounts 30 days past due at month three. An early indicator must itself be validated, by checking that it predicted the final outcome in past data.
The population stability index over time
Lesson 6.4 introduced the population stability index, or PSI, which compares the distribution of the score now with its distribution in development: PSI is the sum over bands of (actual share − expected share) × ln(actual share ÷ expected share). Applied to each input feature, the same calculation is a characteristic stability index, or CSI, which tells you which input moved.
Take a scorecard whose development shares across five score bands, from riskiest to safest, were 8, 18, 30, 26 and 18 per cent. After four months of live use the shares are 12, 20, 30, 22 and 16 per cent. The terms are (0.12 − 0.08) × ln(0.12 ÷ 0.08) = 0.0162, then 0.0021, then 0, then 0.0067, then 0.0024. Their sum is a PSI of 0.027, comfortably below the 0.1 level, which by the usual convention means no significant shift. Notice that the riskiest band has grown by half, from 8 to 12 per cent, and yet the PSI is small. A single number can hide a change that matters to a particular business, which is why the band shares themselves should be on the same report.
Predict first: the PSI is 0.027 in month four. If the same drift continues steadily, with the riskiest band growing by another two or three points each month, will the PSI stay below 0.1 for a long time, or cross it within two or three more months? Decide before reading on.
It crosses quickly. PSI grows roughly with the square of the shift, so equal steady movement accelerates the index. Month five gives 0.057, and month six, with shares of 17, 23, 28, 18 and 14 per cent, gives 0.121. A monitoring plan that checks only whether the index has crossed 0.1 would raise the alarm in month six, but a plan that also tracks the trend would have noticed the climb in months three and four, when it was still 0.007 and 0.027.
When the PSI moves, the next step is to find out why. Compute the CSI of each feature to see which input moved, look at the share of applications by channel, branch and product, and check for pipeline changes. If a new partner channel is sending lower-income applicants, the explanation is data drift and the question is whether the model still ranks them well. If one feature has changed its distribution overnight, suspect the data.
Monitoring outcomes with limits that respect noise
When outcomes do arrive, the natural check is to compare the observed bad rate with the one the model expects. The danger is reacting to noise. A band with a small number of loans has a bad rate that bounces around even when nothing has changed.
The tool is a control band: the range inside which the observed rate should fall about 95 times in 100 if the model is still right. For a band with an expected bad rate of p and n loans, the standard error is the square root of p × (1 − p) ÷ n, and the band is p plus or minus 1.96 standard errors, the same logic as the confidence intervals of Lesson 2.3.
Take the middle score band of the scorecard of Lesson 8.3, with an expected bad rate of 5.33 per cent. In a new cohort it holds 1,200 loans. The standard error is the square root of 0.0533 × 0.9467 ÷ 1,200 = 0.65 percentage points, so the band runs from 5.33 − 1.27 = 4.06 per cent to 5.33 + 1.27 = 6.60 per cent. An observed rate of 5.8 per cent is inside and is no cause for alarm. An observed rate of 7.1 per cent is outside and deserves investigation. The same rate on 120 loans would not be, because the band would then be about 4 points wide on each side. A larger cohort lets you detect smaller changes, which is why monitoring reports show the number of loans and the number of bads beside every rate.
Watching the Gini
The ranking power of the model, measured by the Gini or KS of Lesson 6.2, should also be tracked, cohort by cohort. Here too noise matters. The Gini of the development sample of 5,000 loans with 301 bads was 0.41. A standard approximation for the standard error of an AUC, due to Hanley and McNeil, gives 0.017 for this sample, which makes the standard error of the Gini 2 × 0.017 = 0.034 and a 95 per cent margin of about 0.067. A Gini of 0.38 in a later cohort is within that margin of the original and does not prove deterioration. A Gini of 0.31 is outside it and does.
Two cautions apply. First, a fall in Gini is partly driven by cohort size, so smaller or younger cohorts need wider margins. Second, outcomes are only observed for loans that were approved. If the cutoff changed, the approved population changed too, and the Gini of that narrower group is naturally lower even if the model is no worse, so compare like with like.
Calibration drift and recalibration
A model can keep ranking well while its probabilities drift. Suppose the model predicts an average default of 6.0 per cent and the actual rate in a recent cohort is 7.2 per cent. The ranking, and so the Gini, may be intact. The levels are off, and any decision that uses the level, such as a pricing, provisioning or cost-based cutoff, is now wrong. When ranking is fine and only the level has moved, the remedy is recalibration, which is far cheaper than rebuilding.
The simplest recalibration shifts the model's log-odds by a constant. The predicted odds are 0.06 ÷ 0.94 = 0.0638 and the observed odds are 0.072 ÷ 0.928 = 0.0776. The log of the ratio is ln(0.0776 ÷ 0.0638) = 0.195. Adding 0.195 to every score in log-odds moves the average prediction to the observed rate. On a scorecard, where 20 points double the odds (the "points to double the odds", defined in Lesson 8.3), the same shift is 0.195 × 20 ÷ ln 2 = 5.6 points. Adding 5.6 points, in the right direction, restores the average level. A slightly richer version fits a small logistic regression of the outcome on the model's log-odds in recent data, which adjusts both the level and the slope, the calibration line of Lesson 6.3.
A shift is justified only when the ranking is still sound and the difference is not noise. It also treats the symptoms. If the cause is a change in the data pipeline, fixing the pipeline is the right response, and recalibrating around a bug hides it.
When and how to retrain
Eventually a model needs more than a shift. The choices form a ladder, from cheapest to dearest.
Response · What it does · Use whenInvestigate and fix the data · Repairs a broken feature or pipeline · A feature jumped, the missing rate rose or a unit changed
Recalibrate · Shifts or rescales the output · Ranking is fine and the level is off
Refit · Same features, same method, new coefficients on recent data · Relationships have moved, the structure is still right
Rebuild · New features, new method, new validation · The population or the product has changed in kind
Retire or replace the use · Stop using the model · It cannot be made safe, or the business has changed
Teams decide when to retrain in two ways. A scheduled retrain happens at a fixed interval, such as quarterly or yearly, and is simple to plan and audit. A triggered retrain happens when a monitoring rule fires, such as a PSI above 0.25 or a Gini below an agreed floor. Most mature teams use both: a schedule as a backstop and triggers for the surprises.
Retraining has its own traps.
A narrow view of the data. After deployment you observe outcomes only for approved applicants. The rejected ones have no outcome, so a model refitted on approved loans alone learns a biased picture. Methods called reject inference try to correct this, and a small random approval sample, sometimes called a test and learn group, gives clean data at a known cost.
The model's own feedback. If the model decides who is reviewed or who is offered a product, its past choices shape the data it will next train on. Keep some random exposure.
Windows. Choose the observation and performance windows of Lesson 6.4 on purpose. Recent data is more relevant but its outcomes are immature, and older data has complete outcomes and an outdated population.
Retraining on a bad period. A short extraordinary period, a lockdown or a fraud wave, can mislead a refit. Examine it first, and consider down-weighting or excluding it, with a record of why.
Skipping validation. A retrained model must pass the same validation as the first: out-of-time tests, calibration, stability and documentation. It then runs as a challenger beside the champion, as in Lesson 8.1, before it replaces it.
Finally, decide in advance how severe a signal must be to trigger each response, and who owns the answer. A table of green, amber and red levels, with the action and the person attached to each, prevents both panic and complacency. Too many amber alerts lead to alert fatigue, where people stop reading them, so each threshold should be chosen to fire rarely and to mean something.
python
import numpy as np
import pandas as pd
from sklearn.linearmodel import LogisticRegression
from sklearn.metrics import rocaucscore
def psi(expected, actual, bins=10):
"""Population stability index: bands are cut on the development scores."""
edges = np.quantile(expected, np.linspace(0, 1, bins + 1))
edges[0], edges[-1] = -np.inf, np.inf
e = np.histogram(expected, edges)[0] / len(expected)
a = np.histogram(actual, edges)[0] / len(actual)
e, a = np.clip(e, 1e-4, None), np.clip(a, 1e-4, None) # avoid log(0)
return float(np.sum((a - e) np.log(a / e)))
model = LogisticRegression(maxiter=1000).fit(Xtrain, ytrain)
devscore = model.predictproba(Xtrain)[:, 1]
newscore = model.predictproba(Xvalid)[:, 1]
print("score PSI", round(psi(devscore, newscore), 3))
characteristic stability: which input moved?
csi = {c: psi(Xtrain[c].tonumpy(), Xvalid[c].tonumpy(), bins=5) for c in Xtrain.columns}
print(pd.Series(csi).sortvalues(ascending=False).head(3))
control band for the observed bad rate in a cohort
def controlband(expectedrate, n, z=1.96):
se = np.sqrt(expectedrate (1 - expectedrate) / n)
return expectedrate - z se, expectedrate + z se
lo, hi = controlband(devscore.mean(), len(yvalid))
print(lo, yvalid.mean(), hi)
recalibrate: shift the log-odds so the average prediction matches the observed rate
p = np.clip(newscore.mean(), 1e-6, 1 - 1e-6)
q = np.clip(yvalid.mean(), 1e-6, 1 - 1e-6)
shift = np.log((q / (1 - q)) / (p / (1 - p)))
logit = np.log(np.clip(newscore, 1e-6, 1 - 1e-6) / (1 - np.clip(newscore, 1e-6, 1 - 1e-6)))
newcal = 1 / (1 + np.exp(-(logit + shift)))
print(shift, newcal.mean(), rocaucscore(yvalid, newcal)) # the ranking is unchanged
Common mistakes
Watching only the final metric. Outcomes arrive late, so add input, score and early-indicator monitoring.
Reacting to noise. A band with a few hundred loans swings widely. Use control limits and show the counts.
Using PSI as a verdict. It says the population moved, not that the model failed. Check the outcomes when they arrive.
Trusting a stable PSI. The relationship between inputs and outcome can change without the mix moving.
Recalibrating around a bug. Fix a broken feature before adjusting the model.
Fitting on approved loans only. The rejected applicants carry no outcome. Plan for it with a random sample or reject inference.
Retraining without validation. A new model needs the same tests as the first, and a trial as challenger.
No owner. An alert nobody owns is ignored. Name the person and the action for each level.
Too many alerts. Alert fatigue hides real problems. Set thresholds to fire rarely.
Key takeaways
Models decay because the world moves. Kinds of drift differ: data drift is in the inputs, concept drift is in the relationship, and a data quality change is a bug. Check the data first.
Monitor three layers: operations, inputs and scores, and outcomes. Use early indicators because outcomes arrive late.
PSI = Σ (a − e) × ln(a ÷ e) compares distributions. In the example it rose from 0.027 in month four to 0.121 in month six, and a trend rule catches the climb earlier than a threshold.
A control band, expected rate ± 1.96 × √(p(1 − p) ÷ n), separates signal from noise: 4.1 to 6.6 per cent for an expected 5.3 per cent on 1,200 loans.
Gini needs margins too. With 301 bads the margin is about 0.07, so a fall from 0.41 to 0.38 is not proof and a fall to 0.31 is.
When ranking holds and the level is off, recalibrate: shift = ln(observed odds ÷ predicted odds), 0.195 in the example, or 5.6 scorecard points.
Climb the ladder: fix the data, recalibrate, refit, rebuild. Validate every retrained model and run it as a challenger first.
Check yourself
1. Why does a model monitoring plan need early indicators for a 12-month default outcome?
The final outcome arrives a year after the decision, so a drifting model could go unnoticed for that long. An early indicator, such as 30 days past due at month three, shows trouble sooner.
2. Expected band shares are 25, 25, 25 and 25 per cent, and the actual shares are 40, 30, 20 and 10. Compute the PSI and say how to read it.
The terms are 0.0705, 0.0091, 0.0112 and 0.1374, a total of 0.228. That is a moderate shift, close to the 0.25 mark for a major one.
3. A band has an expected bad rate of 4 per cent and 2,500 loans. What control band applies, and is 5.2 per cent outside it?
The standard error is 0.39 points, so the band is 4 ± 0.77, which is 3.2 to 4.8 per cent. A rate of 5.2 per cent is outside it.
4. A model predicts 3 per cent defaults and 4 per cent occur. What log-odds shift would restore the average, and what is it in points when 20 points double the odds?
ln((0.04 ÷ 0.96) ÷ (0.03 ÷ 0.97)) = 0.298. In points, 0.298 × 20 ÷ 0.693 = 8.6 points.
5. A model's Gini falls from 0.41 to 0.38 on a cohort with about 300 bads. Is that evidence of deterioration?
Not by itself. The margin for a sample this size is about 0.07, so a fall of 0.03 is within noise. Watch the trend and wait for more data.
6. Why is refitting on approved loans only a problem?
Rejected applicants have no outcome, so the refit learns from a selected group and can be biased. A random approval sample or reject inference reduces the bias.
Try this
Take a model from Module 6 and a later slice of the same data to stand in for "live" months. Set aside ninety minutes.
1. Cut the development score into ten bands, compute the share in each band for three later periods, and compute the PSI of each period.
2. Compute the CSI of every input for the latest period, and name the three that moved most.
3. Pick one score band. Compute its expected bad rate, its control band for the latest cohort size and whether the observed rate falls inside it.
4. Compute the Gini of the model on each later period, with an approximate margin from the number of bads. Say which falls are larger than noise.
5. Compare the average predicted probability with the observed rate. Compute the log-odds shift and the effect on the average prediction, and check your arithmetic with the recalibration simulator.
6. Write a monitoring table with a green, amber and red level for PSI, bad rate in band, Gini and missing-value share, and an action and an owner for each red.
7. Write a one-page retraining policy: the schedule, the triggers, the data window, the validation steps and the rule for replacing the champion.
A good result is a monitoring table that a colleague could run without you, with limits that respect noise and an action for every red. The next lesson brings the whole course together in one case: a credit-risk scorecard built from weight of evidence to monitoring plan.
Tags: Data Science Model monitoring Data drift Population stability index Model retraining