Lesson 6.4: Data leakage, performance windows and time-based validation
Learn how the future leaks into models, how observation and performance windows define a fair target, and how to validate across time and track drift with PSI.
By the end of this lesson you will be able to recognise the three families of data leakage, test a model for them before anyone is fooled, define a target with an observation window and a performance window so that it matches the real decision, see why a label measured too early understates risk, validate a model across time instead of across random rows, and compute a population stability index to tell whether the population has shifted. You will see a model whose AUC of 0.97 collapses to 0.76 once the leak is removed and the split is made fair, and work out a PSI by hand from three score bands. You should have read Lessons 5.1, 6.1 and 6.2, since leakage corrupts the validation they describe, and Lesson 3.3 for vintages. The mathematics is a natural logarithm and a handful of ratios.
Leakage is the most expensive mistake in applied machine learning, because it cannot be seen in the numbers that normally tell you a model is good. It produces excellent validation scores, delighted stakeholders and a model that fails the day it is deployed. Overfitting gives you a warning: the gap between training and test error. Leakage gives you no gap at all. The test set is leaking too, so everything looks wonderful, and the only test that matters, the real future, arrives after the money has been spent. For this reason a data scientist's best defence is a habit of suspicion: when a result looks too good, assume a leak and look for it.
What leakage is
Leakage is the use of information in training or validation that would not be available when the model makes a real prediction. The model learns from something it will not have in production, and its validation score reflects that unfair advantage. It comes in three families.
Family · What goes wrong · Typical exampleTarget leakage · A feature contains the answer, or is recorded after the outcome · "Days past due at month 12" used to predict default by month 12
Train-test contamination · Information from the test rows reaches the training process · Scaling, selecting features or encoding on the whole dataset before splitting
Temporal leakage · The model sees the future, because time order was ignored · A random split of time-ordered data, or a feature aggregated over the whole period
### Target leakage
A feature can leak because of what it measures. A bank builds a model to predict which borrowers will default within 12 months of disbursal. Among the available columns is "collections status", a flag set when the collections team contacts a borrower. It is populated for almost every defaulter and almost no one else, so the model discovers that the flag predicts default nearly perfectly. But collections only starts after a borrower has missed payments. At the moment of the lending decision the flag does not exist. In production it would be blank for everyone, and the model's central input would vanish.
The clue is that the field is a consequence of the outcome, not a cause or an early signal. Other examples: a "loan closed early" flag when predicting prepayment; a "cancellation reason" field when predicting cancellation; a refund amount when predicting a return. A rule of thumb for any feature: ask when this value is filled in, and by what process. If it is filled in after the event, or by people reacting to the event, it cannot be used.
Train-test contamination
Here the features are legitimate, and the process is at fault. The most common version is doing a step on the full dataset before splitting it. Lesson 5.1's rule was that scalers, imputers, encoders and bin edges must be learned from training rows only. The same applies to feature selection. There is a well-known demonstration: take 100 rows with 5,000 columns of pure random noise and a random yes-or-no outcome. Choose the 20 columns most correlated with the outcome, using all the rows, then cross-validate a model on those 20. The cross-validated accuracy comes out far above 50 per cent, although there is nothing to learn, because the selection step already used the labels of the rows that were later held out. The cure is to put selection inside each fold of the cross-validation, in a pipeline, so each fold chooses its features from its own training rows.
Other forms: duplicate rows that land in both training and test sets, so the model is tested on cases it has memorised; the same customer, device or household appearing on both sides, which Lesson 6.1's group splitting prevents; and target encoding computed on all the data.
Temporal leakage
When rows are ordered in time, a random split puts later rows in training and earlier rows in testing, so the model is trained on the future to predict the past. If conditions drift, as they always do, the random-split score overstates what the model will do tomorrow. A related form is a feature built over a window that extends past the decision date: "average balance over the year", when the decision was made in March, includes nine months the lender could not have seen.
Warning signs
Leakage hides, but it usually leaves traces.
A score that is too good. An AUC of 0.97 for default prediction, when similar models in the industry reach 0.75 to 0.8, is a leak until shown otherwise.
One feature that dominates. An importance or information value far above the rest, or an IV above 0.5 as Lesson 5.1 noted, is worth auditing.
A feature with a suspicious name or timestamp, such as "status", "flag", "final", "closed" or "resolved".
A drop in performance when a single feature is removed, from excellent to ordinary.
A collapse in production that cannot be explained by drift alone.
A defensive routine is to build a feature availability table: for each input, record the source system, when it is populated and the earliest moment it could be used. Any feature whose availability time comes after the decision time is struck off. It takes an afternoon, and it removes the commonest leaks before modelling begins.
Defining the target: observation and performance windows
Many leaks and many confusions come from a vague target. "Will this borrower default?" is not a data definition. A well-posed prediction problem fixes three things, and credit risk teams name them carefully.
The decision point or snapshot date: the moment the model will be used, such as loan approval.
The observation window: the period before the decision point from which features are drawn, such as the twelve months of credit history up to approval.
The performance window: the period after the decision point over which the outcome is measured, such as the first twelve months after disbursal.
A complete target statement then reads: "Given what we knew at approval, predict whether the borrower reaches 90 days past due at any time in the 12 months after disbursal." Each piece closes a door to leakage. Features come only from before the decision point, and the label comes only from after it. Every feature must be computable from the observation window alone.
Why the length of the performance window matters
Defaults take time to appear. A loan disbursed three months ago has had only three months to go bad, so its label of "not defaulted" is provisional. If you build the target from a short window, you understate the bad rate, and worse, you may label as "good" borrowers who would have defaulted later. This effect is called censoring: the outcome has not had time to be observed.
Predict first: a vintage of loans will eventually reach a 6 per cent default rate. At six months on book, roughly what share of those eventual defaults have appeared: a quarter, a half or nine tenths? Decide before reading on.
The answer is about a half, as the numbers below show. Suppose the final default rate of a vintage is 6 per cent, and defaults appear gradually, so that the cumulative share reached by a given month on book follows a curve that rises quickly at first and flattens. With these assumptions, the observed default rate is about 1.9 per cent at 3 months, 3.2 per cent at 6 months, 4.1 per cent at 9 months, 4.7 per cent at 12 months, 5.4 per cent at 18 months and 5.7 per cent at 24 months.
A model trained on six-month labels would be learning from a population in which more than half of the eventual bad borrowers are still labelled good. The trade-off in choosing the window runs the other way too: a longer window gives more mature labels but uses older loans, and lending conditions may have changed since. A common compromise is a window long enough to capture most of the defaults, say 12 months, with loans younger than that excluded from training or treated as "indeterminate".
Validating across time
Because the future must not inform the past, the right validation for any problem with a time dimension is to split by time. The simplest form is an out-of-time test: train on earlier periods, validate on a later one, and test on a later one still. For loans, split by disbursal date, or vintage.
There is a subtlety. If a label needs 12 months to mature, then loans disbursed in the last 12 months before today do not yet have valid labels. So a realistic arrangement for a December 2026 model might be: train on loans disbursed from January 2024 to December 2024, validate on loans from January to June 2025, and test on loans from July to December 2025, with each group's outcomes observed up to December 2026. Loans disbursed after December 2025 are not yet usable. Some teams also leave a gap between the training and validation periods so that the end of the training period's performance window does not overlap the start of validation, because overlapping outcomes carry related information.
For a model that will be refreshed, rolling-origin validation repeats the process: train on the first block, test on the next, then extend the training window by one block and test on the one after, and so on. The average over the rolling tests shows how the model degrades as the gap between training and use grows.
Scheme · Folds are built · Respects time order · When to use itRandom k-fold · Random rows · No · Data with no time structure, no drift
Group k-fold · Random groups · No · Repeated entities, no time structure
Out-of-time split · By date, one cut · Yes · Any model deployed on future cases
Rolling origin · By date, several cuts · Yes · When you want stability over time
The comparison in the chart above is typical. Even with the leak removed, a random split gave 0.84, while the out-of-time test gave 0.76. The difference is not a flaw in the model. It is the cost of drift, and the out-of-time figure is the honest preview of what deployment will deliver.
Has the population changed? The population stability index
Time-based validation tells you how the model did on yesterday's future. Once the model is live you need to know whether today's cases still resemble the ones it was built on. The standard tool is the population stability index, or PSI. It compares the distribution of a score, or of any feature, in two samples: the expected distribution, from the development data, and the actual distribution, from recent data. Cut the score into bands. For each band, take the share of the expected sample e and the share of the actual sample a, and compute
PSI = the sum over bands of (a − e) × ln(a ÷ e)
Each term is zero when a equals e, and positive otherwise, since the difference and the logarithm have the same sign, so the PSI grows as the two distributions diverge.
A worked example with three score bands. In development, 30 per cent of applicants were in the low band, 40 per cent in the middle and 30 per cent in the high band. In recent data the shares are 20, 40 and 40 per cent. The first band contributes (0.20 − 0.30) × ln(0.20 ÷ 0.30) = (−0.10) × (−0.405) = 0.0405. The middle band contributes 0. The third contributes (0.40 − 0.30) × ln(0.40 ÷ 0.30) = 0.10 × 0.288 = 0.0288. The PSI is 0.0693.
The usual reading, a convention and not a law, is that a PSI below 0.1 means no significant shift, between 0.1 and 0.25 means a moderate shift that deserves investigation, and above 0.25 means a major shift that probably needs action. Here, 0.07 is stable, even though a tenth of the population moved from the low band to the high band. The same calculation applied to individual features is often called a characteristic stability index, and it can point to which input has drifted.
PSI says that the population has moved, not that the model has failed. A shift can be harmless if the relationship between features and outcome is unchanged, or serious if it is not. The ultimate test is performance, measured when outcomes arrive, a matter for Lesson 8.2. But PSI is quick, needs no outcomes, and so gives an early warning in the months before they appear.
A leakage checklist
1. Write the target definition: decision point, observation window, performance window.
2. Build a feature availability table, and strike off anything populated after the decision point.
3. Put every learned step inside the training folds: scaling, imputing, encoding, binning, feature selection.
4. Split by group where one entity has many rows, and by time where there is a time order.
5. Check for duplicates across the splits.
6. Audit the strongest features, and any with an IV above 0.5.
7. Compare random and out-of-time scores. A large gap points to drift or to leakage that follows time.
8. Treat an excellent score as a hypothesis to be tested, not a result.
python
import numpy as np
import pandas as pd
from sklearn.modelselection import TimeSeriesSplit
from sklearn.pipeline import Pipeline
from sklearn.featureselection import SelectKBest, fclassif
from sklearn.preprocessing import StandardScaler
from sklearn.linearmodel import LogisticRegression
from sklearn.metrics import rocaucscore
out-of-time split by disbursal date, using only loans whose 12-month window has closed
df = df[df["disbursaldate"] <= pd.Timestamp("2025-12-31")]
train = df[df["disbursaldate"] < "2025-01-01"]
valid = df[(df["disbursaldate"] >= "2025-01-01") & (df["disbursaldate"] < "2025-07-01")]
test = df[df["disbursaldate"] >= "2025-07-01"]
selection and scaling inside the pipeline, so each fit uses only its own training rows
model = Pipeline([("scale", StandardScaler()),
("select", SelectKBest(fclassif, k=20)),
("clf", LogisticRegression(maxiter=1000))])
rolling-origin validation on date-ordered rows
df = df.sortvalues("disbursaldate")
for tr, te in TimeSeriesSplit(nsplits=4).split(df):
model.fit(df.iloc[tr][features], df.iloc[tr]["bad"])
print(rocaucscore(df.iloc[te]["bad"], model.predictproba(df.iloc[te][features])[:, 1]))
def psi(expected, actual, bins=10):
cuts = np.quantile(expected, np.linspace(0, 1, bins + 1)[1:-1])
e = np.bincount(np.searchsorted(cuts, expected), minlength=bins) / len(expected)
a = np.bincount(np.searchsorted(cuts, actual), minlength=bins) / len(actual)
e, a = np.clip(e, 1e-6, None), np.clip(a, 1e-6, None)
return float(np.sum((a - e) np.log(a / e)))
Common mistakes
Using fields filled in after the outcome. Always ask when a value is populated.
Preprocessing or selecting features before splitting. Everything learned from data belongs inside the training folds.
Random splits on time-ordered data. Split by time, and use a later period for the test.
A label from too short a window. Young loans look good because they have not had time to fail.
No decision point. Without one, features and outcomes blur into each other.
Ignoring duplicates and shared entities. Models memorise them.
Celebrating a spectacular score. Investigate it first.
Mistaking a stable PSI for a stable model. The relationship between features and outcome can change without the population moving.
Key takeaways
Leakage is using information in training or validation that will not exist at prediction time. It comes as target leakage, train-test contamination and temporal leakage, and it makes validation scores look excellent while production fails.
Ask for every feature when and by what process it is populated, and build a feature availability table against the decision point.
A well-posed target has a decision point, an observation window for features and a performance window for the outcome.
Short performance windows censor the label: in the example, six months shows about half of the eventual defaults. Choose a window long enough to mature, and exclude or flag young loans.
Validate across time: out-of-time tests and rolling-origin splits give the honest preview. The same model scored 0.84 on a random split and 0.76 out of time.
PSI = Σ (a − e) × ln(a ÷ e) compares distributions. Below 0.1 is stable, 0.1 to 0.25 moderate and above 0.25 major, and it signals a shifted population, not necessarily a failed model.
Check yourself
1. A model to predict default uses a column "days past due at month 12". What is the problem?
It is recorded after the outcome window closes, so it leaks the answer and will not exist at the decision.
2. Why is selecting the 20 best features on the whole dataset before cross-validation a leak?
The selection uses the labels of the rows that are later held out, so the held-out scores are optimistic. Put selection inside each fold.
3. A vintage will eventually reach 8 per cent default, and 60 per cent of defaults are visible at six months. What default rate do six-month labels show?
0.60 × 8 = 4.8 per cent.
4. Expected band shares are 50, 30 and 20 per cent. Actual shares are 40, 30 and 30 per cent. What is the PSI, and is it a major shift?
(−0.10) × ln(0.8) + 0 + (0.10) × ln(1.5) = 0.0223 + 0.0405 = 0.063. Below 0.1, so no significant shift.
5. A model scores 0.92 on a random split and 0.78 on an out-of-time split. What are two possible explanations?
Drift in the population or relationships over time, or a leak that follows time, such as a feature that reflects later information.
Try this
Use a dataset with a date column and a yes-or-no outcome, such as loans, customers or orders. Set aside forty-five minutes.
1. Write the target definition: decision point, observation window and performance window.
2. List every feature, and for each write the source and when it is populated. Strike off any that arrive after the decision point.
3. Plot the cumulative share of events by time since the decision point, and choose a performance window long enough to capture most of them. Use the window simulator to see what a shorter window would miss.
4. Build a model with a random split and record the AUC. Then rebuild it with an out-of-time split and record the AUC.
5. Put scaling and feature selection into a pipeline and compare with selection done on the whole dataset first.
6. Cut the model's score into ten bands on development data, compute the shares on a later period and calculate the PSI. Use the PSI simulator to check your arithmetic with three bands.
7. Write a short note: the target definition, the features you removed and why, the random and out-of-time scores, and the PSI.
A good result is a note with a precise target, a feature availability table and an out-of-time score that you would be willing to defend. That completes Module 6. You can now split data properly, choose and read the right metrics, set a cutoff from costs and protect a model from leaks. The next module moves to more advanced methods, starting with how to explain what a model has learned.
Tags: Data Science Data leakage Time-based validation Performance window Population stability index