Lesson 3.2: Building a research pipeline
Learn to build a research pipeline: stages from data to report, a one-bar lag against look-ahead, null and truncation tests, and an experiment log.
By the end of this lesson you will be able to lay out a trading research project as a pipeline of separate stages, write each stage so that it gives the same answer every time it is run, prevent look-ahead bias with a one-bar lag, test a pipeline on random data to check that it finds nothing, write a truncation test that catches a leaky signal, and keep a log of every experiment. You will show that a simple momentum rule run on 2,000 days of pure noise earns an annual Sharpe ratio of 0.28 when the pipeline is correct and 2.85 when a signal is allowed to see the day it trades, that the best of 200 random worlds reaches only 0.87, and that eight steps that each carry a 5 per cent chance of an error give only a 66 per cent chance of an error-free result. You should have read Lesson 3.1 on data and Lessons 2.1 to 2.4. The mathematics is a rolling sum, a sign and a Sharpe ratio. This course is education about how markets and research work, not investment advice.
Imagine two researchers with the same idea. The first works in a single long spreadsheet or notebook, pasting data in, editing formulas in place, and keeping the best result on screen. The second writes a small program in which every step is a named function with a clear input and output, runs it from the beginning each time and records every run. After a month, the first cannot say which version of the data produced her best number, or whether she tried the idea twelve ways before this one. The second can rerun any result in a minute, and knows how many ideas she tested. Only the second can tell whether she has found something. A research pipeline is how a quant gets the second kind of workflow.
The stages of a pipeline
Every strategy study has the same shape. Naming the stages helps you build one function for each and find the place where a bug lives.
Stage · Input · Output · A typical bug here1. Load · Files, database, vendor · Raw data, untouched · Wrong time zone, wrong file version
2. Clean and adjust · Raw data · Adjusted, checked data · Unadjusted splits, silently dropped rows
3. Features · Clean data · Variables such as returns, volatility, ratios · A feature that uses the future
4. Signal · Features · A forecast or score at each date · Look-ahead, scaling across the whole sample
5. Positions · Signal, risk rules · Target holdings at each date · Positions applied to the same bar's return
6. Backtest · Positions, prices, costs · Profit and loss per period · Costs left out, fills at impossible prices
7. Evaluate · Profit and loss · Return, volatility, Sharpe, drawdown · Annualising wrongly, no uncertainty
8. Report · All of the above · A page that records what was done · Not saving the settings and the data version
Stages 3 to 6 are the heart of it, and they are where most look-ahead enters. The earlier lessons gave you the pieces. This lesson ties them together.
Principles for a pipeline you can trust
1. One function per stage, with plain inputs and outputs. A function that takes a table and returns a table can be tested alone, reused and replaced. Avoid code that reads and writes hidden global variables.
2. Settings in one place. Put every choice, such as the lookback, the cost and the random seed, in one configuration, and write it out with the results. If a number is hard-coded in the middle of a function, nobody will find it later.
3. Same input, same output. Fix random seeds. Avoid anything that depends on the current time or on the order in which files happen to be read.
4. Never edit raw data; save each stage's output as a new file, as Lesson 3.1 said, so that you can rerun from any stage.
5. Run from the top. A notebook that works only because a cell was run earlier in a different order is not a pipeline. Run the whole thing, fresh, before you trust a result.
6. Version everything. Record the code version, the data version and the configuration with each result. A tool such as git handles the code, and a checksum or a date stamp handles the data.
7. Test the pipeline, not only the strategy. The next sections show two tests that every pipeline should have.
8. Log every experiment, including the ones that failed, for reasons Lesson 3.4 makes precise.
A simple project layout makes these habits natural:
Folder · What it holdsdata/raw · Files exactly as received, never edited
data/clean · Outputs of the cleaning stage, with a version in the name
src · The functions for each stage
tests · The checks on those functions
configs · One file of settings per experiment
results · Output tables and charts, with the settings and version that made them
log · The experiment log, one line per run
## The one-bar lag: the rule against look-ahead
The most damaging pipeline bug is look-ahead bias: using information that was not available when the decision would have been made. The cleanest way to prevent it is a rule you enforce in one place:
A position taken on day t may use only information available at the end of day t − 1 or earlier, and it earns the return of day t.
In code, this is a shift: compute the signal on day t − 1 using data up to and including day t − 1, then trade on day t. If the signal uses the close of day t and you trade at that same close, you must be sure you could really have done so, which is rare. A stricter and safer default is to lag by one bar and make a deliberate exception only when you can justify it.
Here is the damage a missing lag does. We built a toy strategy, the simplest momentum rule, on pure noise. We generated 2,000 days of random returns, each with a standard deviation of 1 per cent a day and no pattern whatever, which is the null world: no strategy can have an edge in it. The signal is the sum of the last 20 daily returns. The rule goes long if the sum is positive and short if it is negative.
Honest pipeline. The signal on day t − 1 sets the position for day t. The Sharpe ratio is 0.28, and the equity curve ends at 1.28 times the start after almost eight years. The standard error of a Sharpe ratio over 7.9 years is about 1 ÷ √7.9 = 0.36, so 0.28 is within one standard error of zero. Nothing was found, which is the right answer.
Same-bar look-ahead. The signal on day t, which includes day t's own return, sets the position for day t. The Sharpe ratio is 2.85, and the equity curve ends at 30.6 times the start. A strategy with no real edge looks like one of the best in the world, only because each day's return is partly inside its own signal.
To get a sense of how far from chance 2.85 is, we repeated the honest pipeline in 200 separate random worlds. The Sharpe ratios averaged 0.05, with a standard deviation of 0.34, and the best of the 200 reached 0.87. A figure of 2.85 on a rule that simple is not skill. It is a bug.
Tests every pipeline should have
The null test
Run the whole pipeline on data in which you know there is nothing to find, such as random returns or real returns with the order shuffled. The answer should be a Sharpe ratio near zero, within its standard error, and about half of the random runs should be positive. If your pipeline reports a strong result on noise, a stage is leaking. This one test, which takes a few lines, catches more errors than any amount of staring at a good result. A related check is to shuffle the signal in time, keeping its values but destroying its timing: a real signal loses its edge when shuffled, so a strong result that survives shuffling has some other source, such as an error in the evaluation. The chart above shows the shuffled result next to the others.
The truncation test
A leak means that the value of a signal at time t depends on data after t. You can test that directly. Compute the signal on the full history. Then compute it again using only the data up to day t, and compare the value on day t. If they differ, the signal used the future. For the honest 20-day sum, the two values match. For a deliberately leaky version, which shifts the signal one day forwards, they do not, and the test fails. Run the truncation test on a few dates for every signal you write, and keep it in the test folder so that every change is checked again.
The experiment log
Every run of the pipeline should add one line to a log: the date, the code version, the data version, the full configuration, the headline results (return, volatility, Sharpe, drawdown, number of trades) and a note on why you ran it. The log has two uses. The first is memory: you can reproduce any result, and see what changed between two of them. The second is honesty. A researcher who tries 50 variants and reports the best has not found what she thinks, and only a log that counts the trials lets you correct for that. Lesson 3.4 shows exactly how to do the correction. Log the failures too. They are the denominator.
Errors multiply: why checks pay
A pipeline has many steps, and an error at any one of them can spoil the result. Suppose each step has a probability e of containing an error that matters. The chance that all k steps are free of error is (1 − e)ᵏ. With e = 5 per cent and eight steps, it is 0.95⁸ = 66 per cent. With twelve steps it is 54 per cent. With a 10 per cent error rate and eight steps it is 43 per cent. In other words, even careful people will usually have at least one error in a long chain, and the longer the chain, the more likely.
Checks work by catching a share of the errors. If your tests, null runs and reviews catch 80 per cent of errors, the effective error rate per step falls from 5 per cent to 1 per cent, and the chance that an eight-step pipeline is clean rises from 66 to 92 per cent. That is the arithmetic behind the habits above.
A complete small pipeline
The code below puts the stages together for the toy strategy. Each stage is a function, the settings sit in one dictionary, and the null test, the shuffled-signal test and the truncation test are at the end. The data is random, so the strategy should find nothing.
python
import numpy as np
import pandas as pd
CONFIG = {"seed": 42, "days": 2000, "dailyvol": 0.01, "lookback": 20, "costbps": 0.0}
def loaddata(cfg):
rng = np.random.defaultrng(cfg["seed"])
return pd.Series(rng.normal(0, cfg["dailyvol"], cfg["days"]), name="ret")
def makesignal(ret, lookback):
return ret.rolling(lookback).sum()
def makepositions(signal, lag=1):
return np.sign(signal).shift(lag) # lag=1: decide yesterday, earn today
def backtest(ret, positions, costbps=0.0):
gross = positions ret
turnover = positions.diff().abs().fillna(0)
return (gross - turnover costbps / 1e4).dropna()
def sharpe(pnl, periods=252):
return pnl.mean() / pnl.std() np.sqrt(periods)
ret = loaddata(CONFIG)
sig = makesignal(ret, CONFIG["lookback"])
honest = backtest(ret, makepositions(sig, lag=1))
peek = backtest(ret, makepositions(sig, lag=0)) # the leak: no lag
null test: shuffle the signal in time, keeping its values
rng = np.random.defaultrng(0)
shuffledsig = pd.Series(rng.permutation(sig.dropna().values), index=sig.dropna().index)
shuf = backtest(ret, makepositions(shuffledsig, lag=1))
print("honest", round(sharpe(honest), 2), "same-bar look-ahead", round(sharpe(peek), 2),
"shuffled signal", round(sharpe(shuf), 2))
print("standard error of a Sharpe over this history", round((1 / (len(honest) / 252)) 0.5, 2))
truncation test: the signal on day t must not change when later data is removed
t = 1500
full = makesignal(ret, CONFIG["lookback"]).iloc[t]
trunc = makesignal(ret.iloc[: t + 1], CONFIG["lookback"]).iloc[t]
print("truncation test passes:", np.isclose(full, trunc))
def leaky(ret, lookback):
return ret.rolling(lookback).sum().shift(-1) # uses the next day
print("leaky signal passes:", np.isclose(leaky(ret, 20).iloc[t], leaky(ret.iloc[: t + 1], 20).iloc[t], equalnan=True))
200 random worlds: what does chance produce?
srs = []
for s in range(200):
r2 = pd.Series(np.random.defaultrng(1000 + s).normal(0, 0.01, 2000))
srs.append(sharpe(backtest(r2, makepositions(makesignal(r2, 20), 1))))
print("200 random worlds: mean", round(np.mean(srs), 2), "sd", round(np.std(srs), 2), "max", round(max(srs), 2))
one line for the experiment log
print({"config": CONFIG, "sharpe": round(sharpe(honest), 2), "trades": int(makepositions(sig).diff().abs().sum() / 2)})
Common mistakes
Trading on the same bar the signal is built from. The pipeline looks magnificent on noise, and fails in live trading.
Normalising with the whole sample. Scaling a feature by its mean and standard deviation over all dates lets the future into the past. Use only data up to each date.
Working in a notebook with hidden state. A result that depends on the order in which cells were run cannot be reproduced.
Hard-coding settings in the middle of a function. The next person cannot tell what the experiment was.
Not setting seeds. A result that changes each time is impossible to debug.
Skipping the null test. A bug that gives a high Sharpe on random data is the cheapest to find and the costliest to miss.
Not logging failed experiments. You will believe your best one, and not know how many you tried.
Mixing research and live code. Keep a clear line, and test that the live version produces the same positions as the research version on the same data.
Treating a good result as proof of a good pipeline. An error that flatters the result is the one you will not look for.
Key takeaways
A research pipeline has stages: load, clean, features, signal, positions, backtest, evaluate and report. Each should be a function with a clear input and output, run from the top with its settings in one place.
The one-bar lag rule: a position on day t may use only information from day t − 1 or earlier. It is a single shift, and it is the main defence against look-ahead.
On 2,000 days of pure noise, a 20-day momentum rule earns a Sharpe ratio of 0.28 with the lag and 2.85 without it, ending at 1.28 and 30.6 times the start. The best of 200 random worlds reached only 0.87.
The null test runs the pipeline on random data and expects a Sharpe near zero within its standard error of 1 ÷ √years. A high result on noise means a bug.
The truncation test compares a signal computed on the full history with one computed on data up to day t. A leaky signal fails it.
Log every run, with the code version, the data version, the settings and the result, and include the failures, since they are the count of trials that Lesson 3.4 needs.
Errors multiply: eight steps with a 5 per cent error rate each are clean 66 per cent of the time, and checks that catch 80 per cent of errors raise that to 92 per cent.
Check yourself
1. A signal uses the closing price of day t and the strategy buys at that closing price. When might this be acceptable, and what is the safe default?
Only if you could really have traded at the close using a price that was known before it, for example a closing auction in which you can submit orders with a known cut-off before the final price is set. Otherwise lag by a bar. The safe default is the one-bar lag.
2. You run your pipeline on shuffled returns and it reports a Sharpe ratio of 1.4 over four years. What does that tell you?
The standard error over four years is 0.5, so 1.4 is nearly three standard errors from zero. The pipeline is finding something in noise, so there is a bug, most likely look-ahead or a data leak.
3. A pipeline has ten steps, each with a 3 per cent chance of an error that matters. What is the chance of no error, and what is it if checks catch 70 per cent?
Without checks: 0.97¹⁰ = 73.7 per cent. With checks the effective rate is 0.9 per cent, so 0.991¹⁰ = 91.4 per cent.
4. Why should a feature such as a z-score be computed with only the data up to each date?
Using the mean and standard deviation of the whole sample means that the value on an early date depends on later data, which is look-ahead bias even though no single line of code looks wrong.
5. What goes into one line of the experiment log, and why log the failures?
The date, code version, data version, settings, headline results and a note on why. Failures count the number of ideas tried, which is needed to judge how impressive the best result really is.
Try this
Set aside forty-five minutes. You need Python and the code above.
1. Run the code. Check that you see a Sharpe of 0.28, 2.85 and −0.22, and a truncation test that passes for the honest signal and fails for the leaky one.
2. Change the random seed in the configuration, and note how the honest Sharpe changes. Repeat for ten seeds and say what range it takes.
3. Change the lookback from 20 to 5 and to 60. Does the honest Sharpe ever look impressive? How many of your attempts would it take to find one above 1.0?
4. Add a transaction cost of 5 basis points per unit of turnover through the costbps setting. How does the honest result move?
5. Write a leaky signal of your own, for example a signal that scales by the maximum of the whole sample, and check that the truncation test catches it.
6. Use the simulator to find how many pipeline steps your own workflow has, and the share of errors your current checks catch. Write down an honest estimate.
7. Write a one-line experiment log entry for each run you did, with the settings and the result.
A good result is a short script that runs from the top, a table of Sharpe ratios on random data that centres on zero, a truncation test that catches your own deliberate leak and a log file. The next lesson turns to what goes into the pipeline: how to build features and signals, and how to judge whether one carries any information.
Read more
Marcos López de Prado's Advances in Financial Machine Learning covers research pipelines, look-ahead and leakage in depth, and Ernest Chan's Quantitative Trading gives a practical view of backtesting pitfalls. For software habits, The Pragmatic Programmer by Andrew Hunt and David Thomas is a general guide to reproducible, testable work, and the documentation for pandas explains the shift and rolling operations used here. For the standard error of a Sharpe ratio, see Andrew Lo's 2002 paper "The Statistics of Sharpe Ratios" in the Financial Analysts Journal.
Tags: Quant Trader Research pipeline Reproducibility Look-ahead bias Experiment log