Lesson 4.1: Backtest design and look-ahead bias
Learn to design a backtest: vectorised and event-driven engines, realistic fills, the biases that flatter results, and checks to run before you trust a number.
By the end of this lesson you will be able to say what a backtest can and cannot tell you, choose between a vectorised and an event-driven engine, list the design decisions to fix before you look at a result, explain how the assumed fill price changes a strategy's Sharpe ratio, name the main biases that flatter a backtest and the test that catches each, and write a truncation test for a whole backtest. You will show that the same simulated strategy has an average Sharpe ratio of 1.54 if it is filled at the close that produced its signal, 0.79 if it is filled at the next open, and 0.43 after costs of 5 basis points per trade, and that a vectorised and an event-driven engine give identical profits when their assumptions match. You should have read Module 3, especially Lessons 3.2 on the pipeline and 3.4 on overfitting. The mathematics is returns, averages and a Sharpe ratio. This course is education about how markets and research work, not investment advice.
Module 3 gave you clean data, a pipeline, signals and a defence against fooling yourself with many tries. Module 4 is about the machine that turns a signal into a track record: the backtest. It is the main tool a quant has for judging an idea before risking money, and it is also the main way quants deceive themselves. The reason is simple. A backtest is a simulation, and a simulation reflects the assumptions built into it. If those assumptions are generous, the result is generous, and nothing on the screen tells you so.
What a backtest is, and what it cannot do
A backtest asks: what would I have earned if I had followed these rules in the past, with the information and the trading conditions that existed then? The answer is a series of simulated profits and losses, from which we compute return, risk and the other measures of Lesson 4.2.
Be clear about its limits.
It cannot prove that a strategy works. The past is one path out of many that could have happened, and a strategy may have been lucky on it. A backtest can reject a strategy, which is its best use, or give modest support to one that has survived every attack you can think of.
It is only as honest as its assumptions. Fills, costs, data and timing each hide ways to flatter the result.
It describes a world in which your trades did not move prices. In reality, trading changes the market, which Lesson 4.4 treats.
It is not independent of the researcher. Every choice made after seeing a result, a new filter, a different window, a changed universe, is a try, as Lesson 3.4 showed.
The right attitude is that of a sceptical auditor: your aim is to find the reason the number is too high.
Two kinds of engine
There are two standard ways to write a backtest.
· Vectorised · Event-drivenHow it works · Computes signals, positions and profits for all dates at once with array operations · Steps through time one bar or one event at a time, keeping the state of the portfolio
Speed · Very fast, thousands of variants in seconds · Slow, but fine for a handful of runs
Realism · Limited: easy for simple daily strategies, awkward for order logic, partial fills and limits · High: can model orders, fills, cash, margin and the order in which things happen
Risk of look-ahead · Higher: one wrong shift leaks the future across the whole array · Lower: the loop only has the past in hand at each step, by construction
Best for · Research and parameter studies · Final validation and anything that will be traded live
The sensible workflow is to research with a vectorised engine and confirm with an event-driven one. If the two disagree, one of them has a bug or the assumptions differ, and either way you have learned something. The code at the end of this lesson writes both for the same strategy and checks that they agree to the last decimal.
An event-driven backtest has five parts, and it helps to name them:
1. A data handler that releases data in time order, so the strategy only ever sees the past.
2. A strategy that turns the data seen so far into target positions.
3. A portfolio that tracks holdings, cash and profit, and turns targets into orders.
4. An execution model that decides at what price and size the orders fill, and what they cost.
5. A performance module that records the results.
The same five parts exist in a vectorised engine, but they are tangled into a few lines of array code, which is why bugs are harder to see.
Fix the design before you look
Decide these things in advance, and write them down.
Decision · What to settleUniverse · Which instruments, selected how, as of each date, including those that later disappeared
Period · Start and end dates, chosen for data quality and not for the result
Data · Which prices, adjusted how, which vendor and version (Lesson 3.1)
Signal timing · When the signal is computed, and from what information
Execution · At what price, and when: the next open, the next close, a volume-weighted price over some period
Position sizing · How a signal becomes a size: equal weights, volatility-scaled, a fraction of Kelly
Rebalancing · How often, and whether trades below a threshold are skipped
Costs · Spread, fees, taxes, slippage and impact (Lesson 4.4)
Constraints · Shorting rules, borrow costs, leverage and position limits
Benchmark · What you will compare against, such as buy-and-hold, cash or a simple version of the idea
Evaluation · The measures, and the out-of-sample plan (Lessons 4.2 and 4.3)
Choosing these after seeing the profit curve is a form of data snooping. The most dangerous decisions are the quiet ones, such as the period and the universe, because they feel like set-up, and they change the answer.
The fill assumption: a small choice with a large effect
Of all the design decisions, the execution assumption is the one beginners most often get wrong. Suppose your signal is computed from today's closing prices. When could you actually have acted on it?
At today's close, using today's close. You cannot: the close is known only when the market shuts, and the order must have been placed before. (There are tricks with closing auctions, but they need an order submitted before the final price is set.)
At tomorrow's open. This is the realistic default for a daily signal: you see the close, decide in the evening and trade when the market opens.
At a later price, such as a volume-weighted average over the first hour, which is more realistic still for a large order.
The difference matters when part of the profit of a signal arrives overnight, in the gap between today's close and tomorrow's open. If you assume you trade at today's close, your simulated self collects that gap. In real life, the gap has happened before you can trade, and the opportunity is gone.
We measured this on a simulated world. A hidden factor, known at the close, predicts the next day's return, and half of its effect lands in the overnight gap and half in the move from the open to the close. We ran the same long-short rule on 50 simulated ten-year histories under three assumptions and averaged the Sharpe ratios.
Execution assumption · Average Sharpe ratio · Return a yearFilled at the close that produced the signal · 1.54 · 24.1%
Filled at the next open · 0.79 · 11.3%
Filled at the next open, with 5 basis points per trade in costs · 0.43 · 6.1%
The rule, the data and the signal are identical. The strategy's Sharpe ratio falls by 72 per cent from the top row to the bottom row, entirely because of what the simulation assumed. Costs alone remove more than 5 percentage points of the 11.3, because the rule turns over 41 per cent of the book every day.
Another way to see the same numbers is to ask where the naive 24.1 per cent a year goes. More than half of it, 12.8 points, is the overnight gap that a real trader cannot collect. Another 5.2 points are costs. Only 6.1 points, a quarter of the headline, is left for the investor.
Try your own assumptions. The simulator takes the return a naive backtest reports and asks how much of it comes from the overnight gap, how often you trade and what each trade costs.
The catalogue of biases
Most backtest failures belong to a short list. Keep it beside you when you review a result.
Bias · What happens · The checkLook-ahead · The signal or the trade uses information not available at the time · Truncation test, one-bar lag, point-in-time data
Survivorship · Delisted names are missing · Use a universe as it was on each date (Lesson 3.1)
Data snooping · The best of many tries is reported · Count the tries and compare with the luck level (Lesson 3.4)
Optimistic fills · Trades at prices you could not have got, or limit orders that always fill · Fill at the next open or later, assume partial fills
Ignored costs · Spread, fees, taxes and impact are left out · Include a realistic cost model and double it as a test (Lesson 4.4)
Liquidity blindness · Trades larger than the market would absorb · Cap order size as a share of daily volume
Short-selling limits · Shorting names that cannot be shorted, or at no borrow cost · Restrict the shorts, charge for borrow
Rounding and lots · Fractional shares and contracts that do not exist · Round to lot sizes and tick sizes
Regime dependence · The result comes from one market condition · Split by period and by volatility regime
Leverage and margin · Positions larger than the capital would allow · Track margin and cash in the portfolio
### Look-ahead in more places than you think
The most dangerous of these is look-ahead bias, and it enters in several ways beyond the plain lag error of Lesson 3.2.
1. Same-bar use. The signal includes the return it then earns.
2. Unpublished data. Earnings, ratings or macroeconomic numbers are used before their release date.
3. Whole-sample normalisation. A z-score, a rank or a threshold is computed over all dates, so each early value depends on later data.
4. Hindsight parameters. A window or a threshold is chosen by looking at the whole history, and then applied from the start of it.
5. Hindsight universe. The list of stocks is today's list, which knows who survived.
6. Calendar leaks. The rebalance dates are chosen after seeing the outcomes, or a "month-end" is defined using data from the following month.
A general defence is the truncation test for a whole backtest. Run the strategy on the full history and record the positions it takes up to a date T. Then run it again on data that stops at T, and compare. If the positions differ on any day, the strategy used something from after T. The test is cheap, mechanical and catches even tiny leaks: the code below shows a whole-sample z-score changing the position on just one day out of 1,500, which no one would find by looking at a profit curve.
Sanity checks before you believe a number
Compare with trivial alternatives. Buy-and-hold, cash, and a random version of the same strategy. If the random version gets close, the result is not special.
Look at where the profit comes from. A few days, one stock or one year should not carry the whole result.
Check turnover and position sizes. An impossible turnover, or a position larger than the day's volume, is a bug or a fantasy.
Reproduce a few trades by hand. Pick five trades and recompute the signal, the fill and the profit from raw data.
Be suspicious of a high Sharpe ratio. Above 2 on a liquid daily strategy is much more often a bug than a discovery.
Stress the assumptions. Double the costs, delay the fills by a day, drop the best month, and see what survives.
The code below builds the simulated world, runs the vectorised and event-driven engines, reproduces the three Sharpe ratios and runs the truncation test.
python
import numpy as np
import pandas as pd
from math import sqrt
def simulate(seed, n=2520, beta=0.12):
"""Daily returns split into an overnight gap and an intraday move. A hidden factor s,
known at the close, predicts half of each next-day piece."""
rng = np.random.defaultrng(seed)
s = np.zeros(n)
e = rng.standardnormal(n)
for t in range(1, n):
s[t] = 0.8 s[t - 1] + 0.6 e[t]
gap, intra = np.zeros(n), np.zeros(n)
for t in range(1, n):
gap[t] = 0.5 beta s[t - 1] / 100 + rng.normal(0, 0.004)
intra[t] = 0.5 beta s[t - 1] / 100 + rng.normal(0, 0.009)
return s, gap, intra
def sharpe(x):
return x.mean() / x.std() sqrt(252)
vectorised engine: a position decided at the close of day t earns day t+1
def vectorised(s, gap, intra, fill="open", costbp=0.0):
pos = np.sign(s)[:-1]
earn = intra[1:] + (gap[1:] if fill == "close" else 0.0)
turnover = np.abs(np.diff(pos, prepend=0))
return pos earn - turnover costbp / 1e4
event-driven engine: step through time, keep the state, fill at tomorrow's open
def eventdriven(s, gap, intra, costbp=0.0):
position, pnl = 0.0, []
for t in range(len(s) - 1):
target = np.sign(s[t]) # known at the close of day t
fee = abs(target - position) costbp / 1e4 # paid when day t+1 opens
position = target
pnl.append(position intra[t + 1] - fee) # then the position earns the intraday move
return np.array(pnl)
s, gap, intra = simulate(0)
a, b = vectorised(s, gap, intra, "open", 5), eventdriven(s, gap, intra, 5)
print("engines agree:", np.allclose(a, b), "| largest difference", float(np.abs(a - b).max()))
rows = []
for seed in range(50):
s, gap, intra = simulate(seed)
rows.append([sharpe(vectorised(s, gap, intra, "close")),
sharpe(vectorised(s, gap, intra, "open")),
sharpe(vectorised(s, gap, intra, "open", 5))])
print("average Sharpe: fill at signal close, next open, next open with 5 bp costs:", np.round(np.mean(rows, axis=0), 2))
truncation test: positions up to day T must not change when later data is removed
r = np.random.defaultrng(2).normal(0.0004, 0.01, 2000)
def honest(r):
return np.sign(pd.Series(r).rolling(20).sum()).values
def leaky(r):
x = pd.Series(r).rolling(20).sum()
return np.sign((x - x.mean()) / x.std()).values # mean and std of the WHOLE sample
T = 1500
for name, f in (("honest", honest), ("whole-sample z-score", leaky)):
full, cut = np.nantonum(f(r)[:T]), np.nantonum(f(r[:T]))
print(name, "passes:", np.arrayequal(full, cut), "| days that differ:", int((full != cut).sum()), "of", T)
Common mistakes
Filling at the price that produced the signal. The close is not available to the decision that uses it.
Reporting a result before fixing the design. Choices made after seeing the profit are tries.
Trusting a vectorised result without a check. One misplaced shift hides in a line of array code.
Leaving out costs because "they are small". At a turnover of 41 per cent a day, 5 basis points cost 5.2 per cent a year.
Assuming limit orders always fill. Fills are rarest when you are right and most frequent when you are wrong.
Normalising or tuning on the whole sample. The future leaks into the past without a visible error.
Using today's universe. The list knows who survived.
Taking positions larger than the market could absorb. The simulated price did not move when you traded, and the real one would.
Looking only at the profit curve. Check the turnover, the concentration and a few trades by hand.
Believing a Sharpe ratio above 2 on a liquid strategy. Search for the bug first.
Key takeaways
A backtest is a simulation under assumptions. It can reject a strategy and give modest support to a survivor, but not prove it.
Research with a fast vectorised engine and confirm with an event-driven one. The two should agree when their assumptions match, as ours did, to the last decimal.
Fix the universe, period, data, timing, execution, sizing, costs, constraints, benchmark and evaluation before looking at results.
The fill assumption alone can change a Sharpe ratio a great deal. The same rule gave 1.54 filled at the signal close, 0.79 at the next open and 0.43 after 5 basis points of costs.
The main biases are look-ahead, survivorship, data snooping, optimistic fills, ignored costs, liquidity blindness, short-selling limits, rounding, regime dependence and leverage. Each has a specific check.
Look-ahead also comes from unpublished data, whole-sample normalisation, hindsight parameters, hindsight universes and calendar leaks. A truncation test on the whole backtest catches even a leak that changes one position in 1,500.
Before believing a result, compare with trivial alternatives, find where the profit comes from, check turnover, reproduce trades by hand and stress the assumptions.
Check yourself
1. A strategy computes a signal from the closing price and the backtest buys at that same closing price. Why is the result overstated?
The close is known only when the market shuts, so you could not have traded at it using the signal. The simulated strategy collects any move between the decision and the real fill, and in a daily strategy that includes the overnight gap.
2. Your vectorised and event-driven backtests give Sharpe ratios of 1.4 and 0.9 for the same strategy. What should you do?
Find out why. Check that the fill price, costs, rebalance timing and position sizes are the same in both, and look for a shifted array in the vectorised code. Do not average them.
3. A strategy turns over 30 per cent of capital a day (one-way) and costs 4 basis points a trade. What is the cost per year?
0.30 × 252 × 4 basis points = 302 basis points, about 3.0 per cent a year.
4. A researcher chooses a 25-day window by looking at which window gives the best profit over the whole ten years, then runs the backtest from the first year. What bias is this?
Look-ahead, in the form of a hindsight parameter, and also data snooping. The window should be chosen on earlier data only, as in walk-forward testing (Lesson 4.3).
5. A backtest shows that 80 per cent of its total profit came from three days. What does that tell you?
The result is fragile and concentrated. Check whether those days are real events or data errors, and see what the strategy earns without them.
Try this
Set aside forty-five minutes. You need Python and the code in this lesson.
1. Run the code. Check that you get "engines agree: True", Sharpe ratios near 1.54, 0.79 and 0.43, and that the truncation test passes for the honest signal and fails for the whole-sample z-score.
2. Change the cost from 5 to 10 basis points and find the Sharpe ratio. Then change the simulation parameter beta to 0.06 and describe what happens to the net result.
3. Add a one-day delay to the fills, so that a signal at the close of day t is traded at the open of day t+2. How much of the Sharpe ratio is left?
4. Write a leaky signal of your own, such as one that uses the maximum of the whole sample, and check that the truncation test catches it.
5. Take one of your own strategies from Module 3 and write its design in the table of this lesson, filling in every row, before you run it.
6. Run it with a naive fill and with a realistic fill and costs. Use the simulator to explain the difference.
7. Write a short note listing the five assumptions in your backtest most likely to flatter it, and a test for each.
A good result is a design table filled in before the run, a vectorised result confirmed by an event-driven one, a truncation test that passes, and a short list of the assumptions you consider weakest. The next lesson takes the profit and loss series a backtest produces and asks how to measure it properly.
Read more
Ernest Chan's Quantitative Trading and Algorithmic Trading describe the practical pitfalls of backtesting, and Marcos López de Prado's Advances in Financial Machine Learning treats backtest overfitting and testing methods in depth. David Aronson's Evidence-Based Technical Analysis is a careful discussion of how data mining misleads. For building an event-driven engine, the open-source projects zipline and backtrader have readable code, and Michael Halls-Moore's online articles on event-driven backtesting in Python are a good tutorial.
Tags: Quant Trader Backtesting Look-ahead bias Event-driven backtest Execution assumptions