Lesson 3.4: Avoiding overfitting: multiple testing and data snooping
Learn why the best of many backtests is usually luck: overfitting, multiple testing, the best Sharpe ratio chance gives, parameter plateaus and a clean holdout.
By the end of this lesson you will be able to explain why a model that fits past data perfectly may predict nothing, work out the chance that at least one of many tests looks significant by luck alone, estimate the best Sharpe ratio that pure chance would give you from a given number of tries, judge a backtest against that figure, tell a broad plateau of parameters from a lucky spike, and design a test that keeps a clean holdout. You will show that a polynomial with nine turns explains 100 per cent of ten noisy data points in-sample and about 2 per cent out of sample, that twenty tests at the 5 per cent level produce at least one false alarm 64 per cent of the time, and that the best of 200 strategies with no edge, tested on four years of data, has an expected Sharpe ratio of about 1.4. You should have read Lesson 3.2 on logging experiments, Lesson 3.3 on signals and Lesson 1.3 on how noisy a Sharpe ratio is. The mathematics is probabilities, powers and the expected largest of a set of random numbers. This course is education about how markets and research work, not investment advice.
This is the lesson that protects you from yourself. The earlier lessons showed you how to find patterns. This one asks the awkward question: how do you know the pattern you found is real? The danger is not that you are careless. It is that searching is itself a source of false discoveries. Look at enough data in enough ways and you will find something impressive, because noise produces impressive-looking patterns at a steady rate. In an old joke a man fires a bullet at the side of a barn, paints a target around the hole and announces that he is a sharpshooter. A backtest with many tuned settings, or the best of many tried ideas, does the same thing: the target is painted after the shot.
Overfitting: fitting the noise
A model has parameters, settings that are chosen to fit data. A moving-average rule has a window length, a stop-loss has a percentage, a regression has coefficients. The more of them you tune, the more closely you can fit any set of numbers, including numbers that are pure noise. A model that fits the noise as well as the pattern is overfitted. It looks excellent on the data it was fitted to, the in-sample data, and disappointing on new data, the out-of-sample data.
Worked example 1. We generated ten data points along a straight line, y = 2x, with random noise added, so the true pattern is simple. We then fitted three polynomials: a straight line (two parameters), a cubic (four) and a ninth-degree polynomial (ten). The fit is measured by R-squared, the share of the variation explained. We then tested each on 1,000 fresh points from the same process.
Model · Parameters · R-squared in-sample (10 points) · R-squared out-of-sample (1,000 new points)Straight line · 2 · 0.69 · 0.76
Cubic · 4 · 0.81 · 0.60
Ninth degree · 10 · 1.00 · 0.02
The ninth-degree polynomial passes through every one of the ten points exactly, a perfect in-sample fit, and then explains almost nothing of new data. It has memorised the noise. The simple line does best out of sample, even though it is the worst in-sample. In-sample fit always rises with complexity. Out-of-sample fit rises and then falls.
The same thing happens with trading rules, only less visibly, because the "polynomial" is hidden in choices: which indicator, which window, which threshold, which markets, which years. Each choice is a knob, and the knobs add up to many degrees of freedom. A rule of thumb is that the number of tuned parameters should be small compared with the number of independent observations, and that every knob should be justified in advance.
Multiple testing: many tries, one winner
The second problem is related but different. Even with no tuning, if you test many ideas, some will look good by chance. Suppose you test a signal and call it significant if the chance of seeing a result this strong under no effect is below 5 per cent. A genuinely useless signal passes about 1 time in 20. Test twenty useless signals independently and the chance that at least one passes is
1 − (1 − 0.05)²⁰ = 1 − 0.358 = 64.2 per cent.
Number of tests · Chance of at least one false positive at 5%1 · 5.0%
5 · 22.6%
10 · 40.1%
20 · 64.2%
50 · 92.3%
100 · 99.4%
The same logic applies to every kind of search: twenty windows for a moving average, a hundred stocks, ten indicators, five markets, twelve holding periods. This is data snooping, or the multiple comparisons problem. The result you report is the best of many, and the best of many is not what a single honest test would give. Remember Lesson 3.2's log: counting the tries is what lets you correct for them.
The simplest fix: a stricter threshold
The oldest remedy is the Bonferroni correction: if you run N tests and want the chance of any false alarm to stay at 5 per cent, require each individual test to pass at 5 per cent divided by N. For twenty tests the cut-off becomes 0.25 per cent, which for a normally distributed test statistic means a t-ratio of about 3.0 instead of 1.96. In 2016 Campbell Harvey, Yan Liu and Heqing Zhu examined hundreds of published factors and argued that, given how many had been tried, a new factor should clear a t-ratio of about 3.0, not 2. Bonferroni is crude, and tends to be too strict when tests are correlated, but it conveys the point: the more you look, the stronger the evidence you must demand.
Check the arithmetic for your own number of tries.
How good a Sharpe ratio does luck give you?
The most useful way to apply this to trading is to ask what the best Sharpe ratio would be by chance alone. Lesson 1.3 showed that a Sharpe ratio estimated from a short history is noisy: its standard error is roughly 1 ÷ √T, where T is the number of years, for a strategy with a true Sharpe near zero. A strategy with no edge, tested on four years of data, has a measured Sharpe ratio that varies around zero with a standard deviation of 0.5.
Now try many strategies. The best of N random numbers, each from the same bell curve, has a predictable size, called the expected maximum. For standard normal numbers it grows slowly with N: about 1.54 standard deviations for ten tries, 2.51 for a hundred, 2.75 for two hundred and 3.24 for a thousand. Multiplying by the standard error of the Sharpe ratio gives the best result luck alone will hand you.
Worked example 2. A researcher tries 200 variants of a strategy, with different windows, thresholds and markets, on four years of daily data. The best has a Sharpe ratio of 1.4, a number that would make any fund manager sit up. The standard error over four years is 1 ÷ √4 = 0.5. The expected maximum of 200 random draws is 2.75 standard deviations, so luck alone should produce a best Sharpe of 2.75 × 0.5 = 1.37. The best result, 1.4, is no better than chance would give. It is not evidence of anything.
We checked this by simulation. We generated 200 strategies with no edge, each 1,008 days of random returns, picked the best, and repeated the whole experiment 200 times. The best Sharpe ratio averaged 1.38. It ranged from 0.95 to 2.29, and in about 42 per cent of the worlds the best of 200 reached 1.4 or more. Seeing a Sharpe of 1.4 in the best of 200 tries is a routine event in a world where nothing works.
Put in your own number of tries, years of data and best result below, and see how far your best result stands above what luck alone would produce.
Deflating a Sharpe ratio
The comparison in worked example 2 is the heart of a method published by David Bailey and Marcos López de Prado in 2014, the deflated Sharpe ratio. It asks how likely it is that the best observed Sharpe ratio exceeds what you would expect from the best of N tries, allowing also for the length of the history and for fat tails and skew in the returns. A simple version of the idea is a haircut: subtract the luck level from the observed Sharpe. In worked example 2 the result is 1.4 − 1.37 = 0.03, essentially zero. The more tries and the shorter the history, the larger the haircut. A published or reported Sharpe ratio means little without saying how many strategies were tried to get it.
Two things make the honest count harder than it looks. First, the tries are not all yours: a signal found by a colleague, or in a paper that tested many variants, carries its own history of searches. Second, tries are not always independent: variants of the same idea are highly correlated, so the effective number of independent tries is smaller than the count, just as for signals in Lesson 3.3. The logic holds regardless, and a rough count is better than none.
Plateaus and spikes
A practical test for overfitting is to look at how the result changes when the parameters change. Suppose you tune a lookback window. Compute the Sharpe ratio for each window from 5 to 100 days.
If the results form a broad plateau, with similar Sharpe ratios for windows of 20, 30 and 40 days, the idea works across a range of settings. A small error in the parameter does not matter, and it suggests a real, robust effect.
If the results form a sharp spike, with a great result at 30 days and nothing at 25 or 35, you have probably found noise. A real effect should not depend on one exact number.
When you choose parameters, pick the centre of the plateau, not the best single point, and, better, average the results across a range of parameters. Averaging over many settings costs a little in the best case and protects you against the spike.
Keeping a clean holdout
The most reliable defence is to test on data that played no part in the search. Divide the history in time:
1. Training set: the data you use to fit and explore, including the choice of features and parameters.
2. Validation set: data used to compare candidates and choose between them.
3. Test set, or holdout: data touched once, at the end, to report the final result.
The word once carries the weight. Every time you look at the holdout and then change something, it leaks into the search, and it becomes a second validation set. A holdout is a consumable. If you have used it three times, it has told you little more than the training data.
Financial data has two features that call for care. Observations are ordered in time, so the split must be by time, with the test set after the training set. Randomly shuffling days into folds lets the future inform the past. And labels often overlap. If you predict the five-day return, consecutive labels share four days, so a training day just before the test period carries information about it. The remedy, described by López de Prado, is to purge the training observations whose labels overlap the test period and to add an embargo, a gap, after it. A more careful scheme is walk-forward testing: fit on a window, test on the next period, roll forward, and repeat. Lesson 4.3 covers it in the backtesting module.
The best test of all is the one the market administers: trade it with small money, or paper trade it, for a period long enough to matter, and compare the result with what the backtest said, taking into account the standard error. A strategy whose live results sit within the backtest's noise band is behaving as expected, and one that falls far below it is overfitted or has decayed.
A discipline that works
1. Write the hypothesis and the story first, before you look at the results, and say what result would make you drop it.
2. Limit the number of tuned parameters, and justify each one in advance.
3. Count every try in the log of Lesson 3.2, including the failures, and correct the result for the count.
4. Demand more for more tries: a higher t-ratio, or a Sharpe ratio above the luck level.
5. Look for plateaus, not peaks.
6. Do robustness checks. Test other markets, other periods, higher costs by half again, slightly different data and slightly different definitions. A real effect survives them.
7. Keep a holdout and use it once.
8. Be most suspicious of the best result. An unusually high Sharpe is much more likely to be a bug or luck than genius.
9. Prefer simple rules. Simplicity is not a guarantee, but every extra knob is another way to fit noise.
10. Size by the doubt. A strategy you are 50 per cent sure of should be run at a fraction of the size you would use if you were certain, which is the Kelly caution of Lesson 2.1 again.
The code below reproduces the main results of this lesson: the overfitting table, the chance of false alarms, the expected best Sharpe from luck, and the simulation of the best of 200 random strategies.
python
import numpy as np
1. overfitting: polynomials fitted to ten noisy points
rng = np.random.defaultrng(3)
x = np.linspace(0, 1, 10)
y = 2 x + rng.normal(0, 0.3, 10)
xt = rng.uniform(0, 1, 1000)
yt = 2 xt + rng.normal(0, 0.3, 1000)
for degree in (1, 3, 9):
coef = np.polyfit(x, y, degree)
r2in = 1 - ((y - np.polyval(coef, x)) 2).sum() / ((y - y.mean()) 2).sum()
r2out = 1 - ((yt - np.polyval(coef, xt)) 2).sum() / ((yt - yt.mean()) 2).sum()
print("degree", degree, "in-sample R2", round(r2in, 2), "out-of-sample R2", round(r2out, 2))
2. chance of at least one false positive
for n in (1, 5, 10, 20, 50, 100):
print(n, "tests:", round((1 - 0.95 n) 100, 1), "%")
print("Bonferroni level for 20 tests at 5%:", 0.05 / 20)
3. expected best Sharpe from luck (Blom-type approximation to the expected maximum of N normals)
def expectedmax(n):
p = (n - 0.375) / (n + 0.25)
return 4.91 (p 0.14 - (1 - p) 0.14)
for n in (10, 100, 200, 1000):
print(n, "tries: expected max", round(expectedmax(n), 2), "times 1/sqrt(5 years) =", round(expectedmax(n) / 5 0.5, 2),
"| four years:", round(expectedmax(n) / 2, 2))
4. simulation: best of 200 random strategies on 4 years of daily data, repeated 200 times
rng = np.random.defaultrng(5)
best = []
for in range(200):
r = rng.normal(0, 0.01, (200, 1008))
sharpe = r.mean(axis=1) / r.std(axis=1, ddof=1) np.sqrt(252)
best.append(sharpe.max())
best = np.array(best)
print("best of 200: mean", round(best.mean(), 2), "range", round(best.min(), 2), "to", round(best.max(), 2),
"share at or above 1.4:", round((best >= 1.4).mean(), 2))
Common mistakes
Judging a strategy by its in-sample fit. The better the fit, the more noise it may have absorbed.
Reporting the best of many tries as if it were the only one. Report how many you made.
Tuning on the holdout. Each look at the test set and each change afterwards spends it.
Shuffling time series into random folds. The future then leaks into the past.
Ignoring overlapping labels. Purge and embargo around the test period.
Picking the spike. A result that depends on one exact parameter is probably luck.
Trusting a high Sharpe over a short history. The standard error is about 1 ÷ √years.
Forgetting the searches of others. Published results and tips already include a lot of selection.
Adding knobs to rescue a failing idea. Every fix after a failed test is another try.
Stopping when the result looks good. Deciding when to stop based on the result is another form of snooping.
Key takeaways
Overfitting means fitting the noise. A ninth-degree polynomial through ten noisy points has an in-sample R-squared of 1.00 and an out-of-sample R-squared of 0.02, while a straight line scores 0.69 and 0.76.
Testing many ideas guarantees some false positives: twenty independent tests at 5 per cent give at least one with probability 64.2 per cent, and a hundred give 99.4 per cent.
A Bonferroni correction divides the significance level by the number of tests. For twenty tests the level is 0.25 per cent, a t-ratio of about 3.
The best of N useless strategies has an expected Sharpe ratio of the expected maximum times 1 ÷ √years. With four years of data, the best of 200 should reach 2.75 × 0.5 = 1.37, and a best result of 1.4 proves nothing.
In simulation, the best of 200 strategies with no edge had a Sharpe ratio averaging 1.38, and reached 1.4 or more in about 42 per cent of repeats.
A broad plateau of parameters is evidence of a robust idea, and a lone spike is typical of a lucky fit. Choose the middle of a plateau.
Keep a holdout untouched until the end and use it once. Split by time, purge overlapping labels, use walk-forward tests and confirm with live or paper trading.
Log every try, set a higher bar for more tries and be most suspicious of your best result.
Check yourself
1. You test 40 independent signals at the 5 per cent level, and none is truly predictive. How many do you expect to look significant, and what is the chance that none does?
Expected number = 40 × 0.05 = 2. The chance that none does is 0.95⁴⁰ = 12.9 per cent.
2. You tried 100 variants on three years of data and the best has a Sharpe ratio of 1.5. The expected maximum of 100 normal draws is 2.5, and the standard error over three years is 0.58. Is 1.5 impressive?
Luck alone gives about 2.5 × 0.58 = 1.45. The best of 1.5 is barely above it, so it is not evidence of an edge.
3. A strategy's Sharpe ratio is 1.2 with a window of 30 days, and 0.1, 0.0 and 0.2 with windows of 25, 35 and 20. What does this pattern suggest?
A spike: the result depends on one exact parameter, which is typical of a fit to noise. A real effect would show similar results at neighbouring windows.
4. You have already looked at your holdout twice and changed the model each time. Is it still a clean test?
No. It has been used as a validation set, and the final result is biased upwards. You need new data, such as a later period or live trading.
5. You predict five-day returns with daily observations, and test on a block of days. What must you do about the days just before the test block?
Purge the training observations whose five-day labels overlap the test period, and consider an embargo gap after it, so that no training label contains test-period returns.
Try this
Set aside forty-five minutes. You can use the code in this lesson, or your own strategy from Lesson 3.3.
1. Run the code, and check that you reproduce the overfitting table, the false-positive table and the best-of-200 simulation.
2. Change the number of strategies in the simulation from 200 to 20 and to 2,000. Plot the average best Sharpe against the number of strategies, and compare with the expected-maximum formula.
3. Change the years of data from 4 to 10. Describe what happens to the luck level and why.
4. Take a signal of your own and a parameter such as a window. Compute the Sharpe ratio for at least ten values of it, and chart the plateau or spike.
5. Count honestly how many variants you tried to reach your best, then use the luck simulator to compare your best with what chance alone would give.
6. Split your data in time into training, validation and test sets of 50, 25 and 25 per cent. Choose parameters on the first two, then look at the test set once, and write down the result.
7. Write a short note: the number of tries, the best result and the luck level, whether the pattern is a plateau, the out-of-sample result and your conclusion about whether the signal is real.
A good result is a chart of the parameter sweep, a comparison of your best Sharpe with the luck level and a holdout result you looked at only once. This lesson completes Module 3. The next module, on backtesting, applies all of this to the design of a proper test: avoiding look-ahead, measuring performance honestly and allowing for costs and capacity.
Read more
David Bailey and Marcos López de Prado's 2014 paper "The Deflated Sharpe Ratio: Correcting for Selection Bias, Backtest Overfitting and Non-Normality" in the Journal of Portfolio Management develops the haircut properly. Campbell Harvey, Yan Liu and Heqing Zhu's 2016 paper "…and the Cross-Section of Expected Returns" in the Review of Financial Studies argues for a higher hurdle for new factors, and Halbert White's 2000 paper "A Reality Check for Data Snooping" in Econometrica gives a formal test. Marcos López de Prado's Advances in Financial Machine Learning explains purged cross-validation and embargoes. Bailey, Borwein, López de Prado and Zhu's 2014 article "Pseudo-Mathematical and Financial Charlatanism: The Effects of Backtest Overfitting on Out-of-Sample Performance" in the Notices of the American Mathematical Society is a short, readable warning.
Tags: Quant Trader Overfitting Multiple testing Data snooping Sharpe ratio