Lesson 4.3: Walk-forward and out-of-sample testing
Learn walk-forward testing: rolling and anchored windows, purging and embargo for overlapping labels, combinatorial splits, and how to read out-of-sample decay.
By the end of this lesson you will be able to run a walk-forward test, choose between rolling and anchored windows, work out how many folds and how much out-of-sample data a design gives you, explain why purging and an embargo are needed when labels overlap, count the splits and paths in a combinatorial scheme, and read the gap between in-sample and out-of-sample results as a measure of overfitting. You will show that a trend rule with a tuned lookback earns an average in-sample Sharpe ratio of 0.56 on random data with no edge and 0.00 out of sample, that on data with a real but slow trend the same procedure falls from 1.63 in sample to 1.09 out of sample, a retention of 67 per cent, and that twelve years of data with a three-year training window and a one-year test window give nine folds and 75 per cent out-of-sample data. You should have read Lessons 3.4 and 4.1. The mathematics is averages, Sharpe ratios and counting combinations. This course is education about how markets and research work, not investment advice.
Lesson 3.4 ended with a rule: keep a holdout and look at it once. That is the right defence, but it has a cost. A single holdout gives you one number from one stretch of history, which is noisy, and once you have looked at it you cannot use it again. Walk-forward testing is the answer to the question this raises: how can you use a lot of the data to judge a strategy honestly, without letting the judging data leak into the fitting?
The idea: always test on what came next
The key principle of any out-of-sample test is that the data used to choose the strategy's settings must come before the data used to judge it. The simplest design splits the history in two: fit on the first part, evaluate on the second. Walk-forward testing repeats that split many times, moving forward through history:
1. Take a training window, for example three years of data. Using only that window, choose the strategy's parameters, such as the lookback of a trend rule, by whatever optimisation you like.
2. Apply those chosen parameters, unchanged, to the test window that follows, for example the next year. Record the profits and losses.
3. Move forward by the length of the test window, and repeat: retrain on the newest three years and test on the year after.
4. When you reach the end, join all the test-window results into one series. That series was produced by a strategy that, at every point, used only past data to choose its settings. It is the closest a backtest can come to what would have happened if you had run the whole procedure live.
Judge the strategy on that joined series, not on the in-sample fits. The training results are only a by-product.
Rolling and anchored windows
There are two ways to define the training window as you move forward.
· Rolling window · Anchored (expanding) windowTraining data · A fixed length, which slides forward · Everything from the start up to now, growing each step
Adapts to change · Quickly, since old data drops out · Slowly, since old data stays
Estimation noise · Higher, because the sample is shorter · Lower, because the sample grows
Best when · Markets change regime and old data is less relevant · The relationship is stable and data is scarce
There is no universally right choice. A short rolling window reacts to change but selects parameters noisily. A long or anchored window selects steadily but may be stale. The honest approach is to fix the design in advance, as Lesson 4.1 advised, and perhaps to compare the two as a robustness check.
The design determines how much out-of-sample evidence you get. With a total history of T years, a training window and a test window, the number of folds is (T − training) ÷ test, and the out-of-sample years are folds × test. With twelve years of data, a three-year training window and a one-year test window, you get (12 − 3) ÷ 1 = 9 folds, and 9 of 12 years, 75 per cent of the data, are used for judging, even though no single year was used to choose its own parameters.
What walk-forward reveals: an experiment
To see what the method does, we ran a controlled experiment. The strategy is a simple trend rule: go long if the sum of the last L daily returns is positive and short if it is negative. The lookback L is chosen from six values, 5, 10, 20, 40, 60 and 120 days. In each fold, the training window picks the L with the best in-sample Sharpe ratio, which is optimisation, and the test window applies it. We built two kinds of world, each with twelve years of daily data.
No edge. Returns are pure random noise. Nothing works, and a trend rule has no reason to earn money.
A real but slow trend. The same noise plus a small drift that changes slowly over time, which a trend rule can follow in part.
We ran 200 worlds of the first kind and 100 of the second, and averaged.
World · Best in-sample Sharpe ratio (average) · Walk-forward out-of-sample Sharpe ratio (average) · Share retainedNo edge · 0.56 · 0.00 · 0%
A real slow trend · 1.63 · 1.09 · 67%
In a world with nothing in it, the optimisation still finds, on average, a lookback with a Sharpe ratio of 0.56 in the training window, because the best of six options on three years of noise looks like something, as Lesson 3.4 predicts. Out of sample it earns exactly nothing, 0.00, because there was nothing to find. The walk-forward test saw through the optimisation. In the world with a real trend, the in-sample number is higher, 1.63, and the out-of-sample one is lower, 1.09, but it is clearly positive: some of the in-sample figure was real, and some was fitted noise.
Two ratios summarise the picture. The walk-forward efficiency, the out-of-sample result divided by the in-sample result, was 0.00 ÷ 0.56 = 0 for the empty world and 1.09 ÷ 1.63 = 0.67 for the world with a trend. A genuine strategy typically keeps a good part of its in-sample performance, perhaps half or more, and an overfitted one keeps little or none. The second signal is the stability of the chosen parameters across folds. If the best lookback jumps from 5 to 120 to 20 from year to year, the optimisation is chasing noise, and a strategy whose chosen setting stays in a narrow range is more believable.
Remember the standard error. The out-of-sample Sharpe ratios in the no-edge experiment had a standard deviation of 0.33 from world to world, and in the trend experiment 0.41. Nine years of out-of-sample data give a standard error of about 1 ÷ √9 = 0.33, so a single walk-forward Sharpe ratio of 1.0 is only three standard errors from zero, and one of 0.5 is not distinguishable from nothing. Walk-forward testing removes the optimisation bias, but it does not make a short history long.
Overlapping labels: purging and the embargo
Walk-forward testing keeps training before testing in time, which stops the future from informing the past. A subtler leak arises when the thing you predict, the label, spans several periods. Suppose your label is the return over the next five days. The label of day t uses the returns of days t + 1 to t + 5. If the test window starts on day 1,000, the training day 996 has a label that runs through day 1,001, which sits inside the test window. The training set therefore contains information about test-period returns, and the test is not clean.
Two remedies come from Marcos López de Prado's work on financial machine learning.
Purging. Remove from the training set every observation whose label overlaps the test window. For a label horizon of h days, that is the h − 1 days just before the test window. For five-day labels, days 996 to 999.
Embargo. When the training data also lies after the test window, as in some cross-validation schemes, drop a further gap of days after the test window, so that serial correlation, for example in volatility, does not carry information back across the boundary. A common choice is about the length of the label horizon.
The cost is small. With five-day labels and a one-year test window of 250 days, purging removes four days and an embargo of five days removes five more, 9 days out of every 250, or 3.6 per cent of the fold. The cost of not doing it is a test that quietly leaks. The same care applies to features that use long windows, such as a 60-day volatility: observations near the boundary share data with the other side.
Combinatorial cross-validation: many paths from one history
Walk-forward testing produces a single path of out-of-sample results, built from the particular order of the history. If that path was lucky or unlucky, you cannot tell. A scheme called combinatorial purged cross-validation gets many paths from the same data. Divide the history into N equal blocks. Choose k of them as the test blocks and train on the rest, with purging and embargo around the boundaries. Do it for every possible choice of k blocks.
The number of train-test splits is the number of ways to choose k blocks from N, C(N, k).
Each block appears in the test set in a share k ÷ N of the splits, so the splits can be reassembled into C(N, k) × k ÷ N complete out-of-sample paths, each covering the whole history.
With N = 6 blocks and k = 2 test blocks, there are C(6, 2) = 15 splits, and 15 × 2 ÷ 6 = 5 paths. Instead of one Sharpe ratio you get five, whose spread tells you how much the result depends on which stretch of history was in the test set. The method trains on data that comes after some of the test blocks, which walk-forward does not, and that suits strategies with no strong dependence on the order of events. Where the order matters, such as when the market structure changed, walk-forward is the safer reading. A careful researcher uses both.
How much decay should you expect in the real world?
Even a careful out-of-sample test does not remove the gap between research and live trading. There are three reasons. The test period is still a period you have seen, and each look at it, followed by a change, wears it out. Markets change: a pattern that was real may weaken or disappear. And other people find and trade the same patterns. A well-known study by David McLean and Jeffrey Pontiff in 2016, of dozens of published stock-return predictors, found that on average the returns were lower after the end of the original samples, by about a quarter, and lower still, by more than half, after the papers were published. The numbers differ across studies and markets, so treat them as an illustration and not a law, but the direction is clear: expect the live Sharpe ratio to be lower than your best out-of-sample number. A common working rule is to haircut a backtest Sharpe ratio by a third to a half before deciding whether a strategy is worth running, and to size by the haircut figure.
The last and strongest test is time itself. Run the strategy on paper, or with a small amount of real money, for a period long enough to say something, and compare the result with what the backtest led you to expect, allowing for the standard error from Lesson 4.2. If the live results sit within the noise band of the backtest, the strategy is behaving as predicted. If they sit far below it, you have either overfitted, or the world has changed.
A workflow that combines the pieces
1. Fix the design in advance: universe, parameters to be tuned and their ranges, training and test windows.
2. Run a walk-forward test, with purging and an embargo where labels overlap.
3. Report the joined out-of-sample series, not the in-sample fits. Give the Sharpe ratio, its standard error, the drawdown and the stability of the chosen parameters.
4. Compare in-sample and out-of-sample results. A large gap means overfitting.
5. Run robustness checks: a different window length, anchored and rolling, other markets, higher costs.
6. Keep a final holdout, untouched by any of the above, and use it once.
7. Paper trade or trade small, and compare with the expected range.
8. Size by the haircut Sharpe ratio, not by the best number you saw.
The code below builds the two kinds of world, runs the walk-forward test on each and counts the combinatorial splits.
python
import numpy as np
from math import sqrt, comb
LOOKBACKS = (5, 10, 20, 40, 60, 120)
def sharpe(x):
sd = x.std(ddof=1)
return x.mean() / sd sqrt(252) if sd > 0 else 0.0
def strategypnl(r, lookback):
"""Trend rule: long if the sum of the previous lookback returns is positive, else short."""
cs = np.concatenate([[0.0], np.cumsum(r)])
signal = np.full(len(r), np.nan)
signal[lookback:] = cs[lookback:len(r)] - cs[:len(r) - lookback] # known at the start of day t
return np.nantonum(np.sign(signal) r)
def makeworld(seed, years=12, edge=False):
rng = np.random.defaultrng(seed)
n = years 252
r = rng.normal(0, 0.01, n)
if edge: # a slowly varying drift, which a trend rule can follow
drift = np.zeros(n)
for t in range(1, n):
drift[t] = 0.97 drift[t - 1] + rng.normal(0, 0.0005)
r = r + drift
return r
def walkforward(r, train=3, test=1):
insample, outpieces = [], []
n = len(r)
for start in range(train 252, n - test 252 + 1, test 252):
window = slice(start - train 252, start)
evaluation = slice(start, start + test 252)
bestlb, bestsr = None, -9.0
for lb in LOOKBACKS: # optimise on the training window only
pnl = strategypnl(r[: start + test 252], lb)
if sharpe(pnl[window]) > bestsr:
bestlb, bestsr = lb, sharpe(pnl[window])
insample.append(bestsr)
outpieces.append(strategypnl(r[: start + test 252], bestlb)[evaluation]) # then test it
return np.mean(insample), sharpe(np.concatenate(outpieces)), len(insample)
for label, edge, worlds in (("no edge", False, 200), ("with a trend", True, 100)):
rows = np.array([walkforward(makeworld(seed, edge=edge)) for seed in range(worlds)])
print(f"{label}: best in-sample Sharpe {rows[:, 0].mean():.2f} | walk-forward out-of-sample Sharpe {rows[:, 1].mean():.2f}"
f" (sd {rows[:, 1].std():.2f}) | folds {int(rows[0, 2])}")
print("12 years, train 3, test 1:", (12 - 3) // 1, "folds,", 9 / 12, "of the data is out-of-sample")
N, k = 6, 2
print("combinatorial splits", comb(N, k), "| out-of-sample paths", comb(N, k) k // N)
purging: with h-day labels, drop the last h - 1 training days before the test window
h, teststart = 5, 1000
print("training days whose 5-day labels reach the test window:", list(range(teststart - (h - 1), teststart)))
Common mistakes
Reporting the in-sample fits. The training results always look good. Judge the joined out-of-sample series.
Choosing the window lengths after seeing the result. The design is a parameter too, and tuning it is a try.
Using the same test period many times. Each look, followed by a change, turns it into training data.
Ignoring overlapping labels. A five-day label reaches four days into the next period. Purge and embargo.
Using random folds for time series. The future then helps to predict the past.
Re-optimising too often on too little data. The chosen parameters become noise.
Reading one walk-forward Sharpe ratio as precise. Nine years give a standard error of about 0.33.
Expecting the out-of-sample figure to hold live. Allow for decay from selection and from others trading the same idea.
Forgetting costs in the test windows. The out-of-sample series must be net of the costs of Lesson 4.4.
Key takeaways
Walk-forward testing fits on a training window, tests on the window after it, rolls forward and joins the test results into one series that never used its own period to choose its settings.
A rolling window adapts faster but is noisier, and an anchored window is steadier but stale. The number of folds is (T − training) ÷ test: twelve years with three-year training and one-year tests gives nine folds and 75 per cent out-of-sample data.
In our experiment, tuning a trend rule on data with no edge gave an in-sample Sharpe ratio of 0.56 and an out-of-sample ratio of 0.00. With a real slow trend the figures were 1.63 and 1.09, a walk-forward efficiency of 67 per cent.
The gap between in-sample and out-of-sample results is a measure of overfitting, and parameters that jump around from fold to fold are a warning.
Overlapping labels leak across the split. Purge the h − 1 training days before the test window, and embargo a gap after it.
Combinatorial purged cross-validation gives many paths: six blocks with two used for testing give 15 splits and 5 paths.
Expect decay: published predictors have in studies lost about a quarter of their return out of sample and more than half after publication. Haircut a backtest Sharpe ratio by a third to a half, and confirm with paper or small live trading.
Check yourself
1. You have 8 years of data, a 2-year training window and a 6-month test window. How many folds do you get, and how many years of out-of-sample data?
Folds = (8 − 2) ÷ 0.5 = 12. Out-of-sample years = 12 × 0.5 = 6, which is 75 per cent of the history.
2. A strategy's in-sample Sharpe ratio is 2.0 and its walk-forward out-of-sample Sharpe ratio is 0.3. What does that suggest?
A walk-forward efficiency of 15 per cent, so most of the in-sample performance was overfitting. The strategy is very likely noise or heavily over-tuned.
3. You predict returns over the next 10 days. Your test window starts on day 2,000. Which training days must be purged?
Days 1,991 to 1,999, the nine days (h − 1) before the test window, because their 10-day labels reach into the test window.
4. You divide a history into 8 blocks and use 3 as test blocks in each split. How many splits and paths are there?
Splits = C(8, 3) = 56. Paths = 56 × 3 ÷ 8 = 21.
5. A walk-forward test on 5 years of out-of-sample data gives a Sharpe ratio of 0.9. Is it distinguishable from zero?
The standard error is about 1 ÷ √5 = 0.45, so the t-ratio is 0.9 ÷ 0.45 = 2.0, just at the threshold, with a 95 per cent range of about 0.02 to 1.8. It is suggestive but not conclusive.
Try this
Set aside forty-five minutes. You need Python and the code of this lesson.
1. Run the code. Check that you see in-sample figures of 0.56 and 1.63 and out-of-sample figures near 0.00 and 1.09.
2. Change the training window from 3 years to 1 and then to 6. How do the in-sample and out-of-sample Sharpe ratios change, and why?
3. Change the lookback set to include only two values, 20 and 60. Does the gap between in-sample and out-of-sample results shrink? What does that say about the number of knobs?
4. Replace the rolling window by an anchored window, which starts from the first day each time, and compare.
5. Record the lookback chosen in each fold for one world and plot it. Is it stable?
6. Apply the same walk-forward design to a strategy from Lesson 3.3 or one of your own, with a realistic cost, and report the joined out-of-sample series with its Sharpe ratio, standard error and maximum drawdown.
7. Write a short note: the design, the in-sample and out-of-sample Sharpe ratios, the walk-forward efficiency, the stability of the parameters and your estimate of the Sharpe ratio you would expect live.
A good result is a joined out-of-sample series with its uncertainty, a table of chosen parameters per fold and a plain statement of how much of the in-sample result survived. The next lesson turns to the cost side of every test in this module: what it takes to trade, how costs and market impact scale with size, and how much money a strategy can really run.
Read more
Marcos López de Prado's Advances in Financial Machine Learning describes purged cross-validation, embargoes and the combinatorial scheme in detail. Robert Pardo's The Evaluation and Optimization of Trading Strategies is a classic practical account of walk-forward analysis. David McLean and Jeffrey Pontiff's 2016 paper "Does Academic Research Destroy Stock Return Predictability?" in the Journal of Finance measures how returns decay out of sample and after publication. For the statistics behind the standard errors used here, see Andrew Lo's 2002 paper "The Statistics of Sharpe Ratios".
Tags: Quant Trader Walk-forward testing Out-of-sample Cross-validation Overfitting