Lesson 6.1: Train, validation and test splits, and cross-validation
Learn why a model must be judged on data it has not seen, how to split data into training, validation and test sets, and how cross-validation works.
By the end of this lesson you will be able to split data into training, validation and test sets and explain what each is for, say how uncertain a test score is from the size of the test set, run k-fold cross-validation by hand on five scores, choose between the common validation schemes, and avoid the quiet mistake of tuning a model until it fits the test set. You will compute how wide the margin on an 85 per cent accuracy is for 100, 1,000 and 5,000 test rows, see why picking the best of twenty useless models produces a score about nine points above chance, and learn when plain random splits are wrong. You should have read Lesson 4.4, since this lesson is the practical answer to the overfitting problem it posed, and Lesson 2.3 for margins of error. The mathematics is averages, square roots and the standard error of a proportion.
Everything in the last three modules assumed that you could tell a good model from a bad one. This module shows how, and it starts with the most important rule in applied machine learning: a model must be judged on data it was not trained on. A model's performance on the rows it learned from is a flattering fiction, since it has had the chance to memorise them. What a business cares about is performance on the next loan, the next customer, the next month. Validation is the discipline of estimating that honestly, before the model is deployed and the real answer arrives.
Three sets, three jobs
The standard approach divides the data into three parts.
The training set is what the model learns from: coefficients, tree splits, weights.
The validation set is used to make choices: which algorithm, which features, which settings, when to stop. You may look at its score as often as you like.
The test set is kept locked away until the very end. It is used once, to report how the final model is expected to perform on new data.
A typical split is 60 per cent for training, 20 for validation and 20 for testing, though the proportions depend on how much data you have. With a million rows, 1 per cent held out is plenty. With two thousand rows, you will want to use cross-validation, described below, in place of a fixed validation set.
Predict first: you train a model, check it on the validation set, adjust it, check again, and repeat thirty times. You finally report the score on the validation set as the expected real-world performance. Will the real performance be higher than, equal to or lower than the number you report? Decide before reading on.
It will usually be lower. Each time you adjusted the model in response to the validation score, a little of that validation set leaked into your choices, and thirty rounds of this tunes the model to the validation set's quirks. The reported number is optimistic. The test set exists precisely to catch this: since you never used it to make a choice, its score is an unbiased estimate. And it only stays unbiased if you use it once. If you look at the test score, dislike it, change the model and test again, the test set has become another validation set, and you have lost your honest estimate.
Set · Used for · How often you may look · Score reflectsTraining · Fitting the model · Constantly · Memory, optimistic
Validation · Choosing models and settings · Many times · Slightly optimistic, because of the choices
Test · Final report · Once · Honest estimate of future performance
## Doing the split properly
A split sounds trivial, and several details separate a sound one from a flawed one.
Shuffle before splitting, unless the rows are ordered in time, in which case see Lesson 6.4. A file sorted by date or by customer would otherwise put all of one kind in the test set.
Stratify on the outcome. With a rare outcome, a plain random split can give the test set a very different share of defaulters from the training set. Suppose the default rate is 6 per cent and the test set has 100 rows. The expected number of defaulters is 6, but the standard deviation of the count is √(100 × 0.06 × 0.94) = 2.4, so you will often see 3 or 10. A model's recall computed on 3 defaulters is meaningless. Stratified splitting keeps the share of defaulters the same, 6 per cent, in every part.
Keep related rows together. If one customer has several loans, putting some in the training set and some in the test set lets the model recognise the customer, and its test score is inflated. Split by customer, not by row. The same applies to repeated measurements of one device, one patient or one household. This is called group splitting.
Fix the random seed, so that you or a colleague can reproduce the split exactly.
Do all learning from the training set alone. Scaling constants, imputation values, encodings and bin edges, as Lesson 5.1 insisted, must be computed from training rows and merely applied to the others.
How certain is a test score?
A test score is itself an estimate, with its own sampling error, because the test set is a sample. If a model scores 85 per cent accuracy on 200 test rows, how much would you trust that number? Lesson 2.3 gave the margin of error for a proportion: 1.96 × √(p × (1 − p) ÷ n). For p = 0.85 and n = 200, that is 1.96 × √(0.1275 ÷ 200) = 1.96 × 0.0253 = 0.050. The honest statement is 85 per cent, give or take 5 points. With 100 test rows the margin is 7.0 points, with 500 it is 3.1, with 1,000 it is 2.2, and with 5,000 it is 1.0.
This has a practical consequence. If model A scores 85 per cent and model B scores 83 per cent on a test set of 500 rows, the difference of 2 points is well inside the margin of 3 points for each, and you cannot say that A is better. Comparing models on a small test set is mostly comparing noise. When differences are small, compare the models on the same rows and test the paired differences, or use cross-validation to gather more evidence.
Count the cases that matter, not the rows
For rare outcomes, the number of rows in the test set overstates how much evidence you have. Take a 6 per cent default rate and a test set of 1,000 rows. The test set has only about 60 defaulters, and any metric about defaulters, such as recall, rests on those 60. If recall is 70 per cent, its margin of error is 1.96 × √(0.7 × 0.3 ÷ 60) = 1.96 × 0.059 = 11.6 points. A recall of 70 per cent could easily be 58 or 82. To bring the margin to about 5 points you would need roughly 300 defaulters, which at a 6 per cent rate means a test set of about 5,000 rows. So when you plan a split, ask how many cases of the rarest outcome each part will contain, and aim for at least a few hundred in the test set where you can. If the data cannot supply that many, say so in the report, and lean on cross-validation, which pools more cases into the estimate.
The same logic applies to the training set. A model that must learn from 40 defaulters will be unstable whatever the total number of rows, and the guidance from Lesson 4.4, of 10 to 20 cases of the rarer outcome for each predictor you estimate freely, is a useful sanity check on how many features you can afford.
Cross-validation: using the data twice over
When data is limited, setting aside 20 per cent for validation is wasteful and noisy. K-fold cross-validation makes better use of the rows. It splits the training data into k equal parts called folds. For each fold in turn, it trains on the other k − 1 folds and scores on the held-out fold. Every row is used for validation exactly once and for training k − 1 times, and you get k scores whose average is a more stable estimate than a single split gives.
Here are five scores from 5-fold cross-validation of a classifier: 0.81, 0.84, 0.79, 0.83 and 0.85. The mean is 0.824. The deviations from the mean are −0.014, 0.016, −0.034, 0.006 and 0.026. The squares add to 0.00232, the sample variance is 0.00232 ÷ 4 = 0.00058, and the standard deviation is 0.024. The standard error of the mean is 0.024 ÷ √5 = 0.011. So you would write: cross-validated accuracy of 82.4 per cent, with fold-to-fold variation of about 2.4 points. The spread between folds is also information. A model whose fold scores range from 0.60 to 0.95 is unstable, whatever its average.
Common variants answer particular needs.
Scheme · How it works · Use it when · Watch out forHold-out · One split into train and validation · Lots of data · Noisy with small data
K-fold · k folds, each held out once · The standard choice, k of 5 or 10 · Costs k fits
Stratified k-fold · Each fold keeps the outcome share · Classification, especially rare outcomes · Still random
Repeated k-fold · K-fold repeated with new shuffles · Small data, need a steadier estimate · Costs more fits
Leave-one-out · Each row is its own fold · Very small data · n fits, high variance
Group k-fold · Rows of a group stay in one fold · Repeated customers, devices, patients · Needs a group column
Time-series split · Train on the past, validate on the next block · Ordered data · Lesson 6.4
Cross-validation has a cost: you fit the model k times, and if you also search over settings, you multiply again. Fitting a model for 2 minutes, with 5 folds and 50 candidate settings, takes 5 × 50 × 2 = 500 minutes, over eight hours. Budget for it, and use a cheaper search first.
Tuning without fooling yourself
Cross-validation is usually used to choose settings: the number of trees, the strength of a penalty, the depth. You pick the setting with the best average score. But this reintroduces optimism: the best of fifty scores is, on average, higher than the true quality of the winning setting, because the winner was partly selected for lucky noise.
The effect is easy to see by imagining models that have no skill at all. Suppose each candidate has a true accuracy of 50 per cent, a coin flip, and you score each on 100 test rows. Each score varies with a standard deviation of √(0.25 ÷ 100) = 5 points. One candidate scores around 50, as you would expect. The best of 20 such candidates, however, scores on average 1.87 standard deviations above the mean, which is about 9 points higher, or 59 per cent. The best of 100 scores about 12.5 points higher. You would report a model that looks 59 per cent accurate when nothing in it works.
The cure is to keep a final judge that did not take part in the selection. When you tune with cross-validation on the training data, the test set is that judge. For the most careful work, with small data, there is nested cross-validation: an inner cross-validation chooses the settings, and an outer cross-validation measures how well the whole procedure, including the choosing, performs on data that played no part in it. It costs a great deal more computation, and it gives an honest estimate of a tuned model on limited data.
A further discipline is to count how many things you have tried. The score of the best of fifty experiments is not the score of a single pre-planned experiment, and a report should say how many models, feature sets and settings were compared. This is the multiple testing trap of Lesson 2.3 in a new costume.
A validation workflow
1. Set aside the test set first, stratified and grouped if needed, and do not touch it.
2. Build a simple baseline and score it by cross-validation on the training data. Everything else must beat it.
3. Develop and compare models by cross-validation on the training data, with all preprocessing inside the cross-validation loop.
4. Tune with a limited, sensible search, noting how many settings you tried.
5. Check stability: the spread across folds, and whether the ranking of models is the same on different seeds.
6. Refit the chosen model on all the training data.
7. Score once on the test set, and report the score with its margin of error.
8. Do not go back. If the test score disappoints, the answer is to collect more data or rethink the problem, not to tune on the test set.
python
import numpy as np
from sklearn.modelselection import (traintestsplit, StratifiedKFold, GroupKFold,
crossvalscore, GridSearchCV)
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.pipeline import makepipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linearmodel import LogisticRegression
1. lock away the test set, stratified on the outcome
Xtrain, Xtest, ytrain, ytest = traintestsplit(
X, y, testsize=0.2, stratify=y, randomstate=42)
2. baseline, with scaling inside the pipeline so each fold fits its own scaler
base = makepipeline(StandardScaler(), LogisticRegression(maxiter=1000))
cv = StratifiedKFold(nsplits=5, shuffle=True, randomstate=42)
scores = crossvalscore(base, Xtrain, ytrain, cv=cv, scoring="rocauc")
print(scores.mean(), scores.std(ddof=1))
group-aware folds when one customer has several rows
gscores = crossvalscore(base, Xtrain, ytrain, groups=customeridtrain,
cv=GroupKFold(nsplits=5), scoring="rocauc")
4. a modest search, then 7. a single look at the test set
search = GridSearchCV(HistGradientBoostingClassifier(),
{"learningrate": [0.03, 0.1], "maxdepth": [3, 5]},
cv=cv, scoring="rocauc").fit(Xtrain, ytrain)
print(search.bestparams, search.bestscore)
print(search.score(Xtest, ytest)) # the one and only test-set score
Common mistakes
Reporting the training score. It measures memory, not skill.
Tuning on the test set. Every look at it spends its honesty. Look once.
Preprocessing before splitting. Scalers, imputers and encoders must be fitted inside the training folds.
Random splits with related rows. Use group splitting when a customer, device or household has many rows.
Not stratifying a rare outcome. The test set can end up with a handful of positives.
Comparing models on a tiny test set. Differences smaller than the margin of error are noise.
Ignoring the spread across folds. A good mean with a wide spread signals an unstable model.
Forgetting the count of comparisons. The best of many is optimistic. Count what you tried.
Randomly splitting time-ordered data. Use time-based validation, as the last lesson of this module shows.
Key takeaways
A model must be judged on data it has not seen. Use a training set to fit, a validation set to choose, and a test set to report, looked at once.
Stratify on a rare outcome, keep related rows together, shuffle unless the data has a time order, and fix the seed.
A test score has a margin of error: 1.96 × √(p(1 − p) ÷ n). At 85 per cent it is about 7 points on 100 rows and 1 point on 5,000.
K-fold cross-validation uses every row for validation once, and gives k scores whose mean and spread describe performance and stability. It costs k fits.
Choosing the best of many models is optimistic. The best of 20 useless models looks about 9 points above chance on 100 rows.
Keep a final test set out of every choice, count what you tried, and use nested cross-validation for tuned models on small data.
Check yourself
1. Why should you not use the test set to choose between two models?
Choosing with it makes it part of the selection, so its score becomes optimistic and no longer an honest estimate.
2. A model scores 90 per cent on 400 test rows. What is the approximate 95 per cent margin of error?
1.96 × √(0.9 × 0.1 ÷ 400) = 1.96 × 0.015 = 0.029, about 2.9 points.
3. In 5-fold cross-validation the scores are 0.70, 0.74, 0.72, 0.76 and 0.78. What are the mean and the sample standard deviation?
Mean 0.74. Deviations −0.04, 0, −0.02, 0.02, 0.04, squares sum to 0.004, variance 0.001, standard deviation 0.032.
4. Customers have several loans each. Why is a random row split a problem?
The same customer can appear in both training and test sets, so the model can recognise them and the test score is inflated. Split by customer.
5. Tuning with 10 folds and 20 settings takes 3 minutes per fit. How many fits and how many hours is that?
200 fits, 600 minutes, 10 hours on one core.
Try this
Use a dataset with an outcome column and a few hundred rows or more. Set aside forty-five minutes.
1. Split off a 20 per cent test set with a fixed seed and stratification. Write down how many positives are in each part. Do not look at the test set again until step 7.
2. Fit a simple baseline and score it with 5-fold cross-validation. Record the mean and the standard deviation across folds.
3. Fit two more models, such as a regularised logistic regression and a boosted tree, with preprocessing inside a pipeline. Compare their cross-validated scores and their spreads.
4. Use the test-set margin simulator to work out how many test rows you would need to tell a 2-point difference from noise.
5. If your data has a repeating entity such as a customer, repeat the cross-validation with group folds and note how the score changes.
6. Tune one model over at least ten settings. Record how many you tried and the gap between the best cross-validated score and the average.
7. Score the final model on the test set once, and report the score with its margin of error.
8. Write a short note: baseline against model, the stability of the folds, the count of comparisons and the final test score.
A good result is a note in which every score comes with an uncertainty, and the test score appears once at the end. The next lesson turns to what the score should be: the metrics that matter for classification, and why accuracy is often the worst one.
Tags: Data Science Model validation Cross-validation Train test split Model evaluation