Lesson 3.1: Market data: cleaning, adjustments and survivorship bias
Learn to prepare market data for research: bars, adjusting for splits, bonuses and dividends, cleaning bad ticks, and avoiding survivorship bias.
By the end of this lesson you will be able to name the main kinds of market data and what each is good for, adjust a price series for a split, a bonus issue and a dividend so that returns are correct, find and handle bad prints and gaps, measure the survivorship bias in a backtest that uses only today's winners, and explain why data must be recorded as it was known at the time. You will show that a share that splits one into five shows a fake fall of 80 per cent in unadjusted data, that a single bad tick can sit more than 6,000 robust standard deviations from its neighbours, and that a universe of 100 stocks in which 20 later failed overstates the average return by 4.8 percentage points if you look only at the 80 that survived. You should have read Module 2, especially Lesson 1.3 on returns and Lesson 2.4 on stationarity. The mathematics is ratios, medians and weighted averages. This course is education about how markets and research work, not investment advice.
The first two modules gave you the market and the mathematics. A third thing decides whether your research is worth anything: the data. Every result a quant produces is a function of the numbers fed in, and a famous saying applies: garbage in, garbage out. The garbage is rarely obvious. It is a price that has not been adjusted for a split, a missing day filled in with the wrong value, a list of companies that quietly leaves out the ones that went bust. Each of these makes a strategy look better than it will be in live trading, and none of them shows an error message. Professional firms spend more effort on data than on models for this reason.
What kinds of data there are
Market data comes at different levels of detail, and the level you need depends on the strategy.
Data · What it is · Typical use · Common problemsTick data · Every trade, and often every quote change, with a timestamp · Execution research, market making, microstructure · Huge volumes, bad prints, crossed quotes, clock errors
Bars (OHLCV) · Open, high, low, close and volume for each minute, hour or day · Most research, from intraday to daily strategies · Wrong time zones, missing bars, unadjusted prices
Order book snapshots · The queue of bids and offers at several price levels · Short-horizon prediction, execution · Storage, gaps, depth that was not real
Reference data · Names, identifiers, lot sizes, tick sizes, trading calendars · Everything · Tickers change, get reused or are dropped
Corporate actions · Splits, bonuses, dividends, mergers, rights issues · Adjusting prices, computing returns · Missing or late records, vendors that disagree
Fundamentals · Revenue, profit, balance sheet items · Value and quality signals · Restatements, announcement delays
Alternative data · Web traffic, satellite images, card spending, text · Signals beyond prices · Short history, bias in who is covered
A bar summarises the trades in an interval: the open is the first price, the high the highest, the low the lowest, the close the last, and the volume the total traded. Building bars from ticks is simple, and it has traps. Exchange timestamps are in local time, and a global database may store them in UTC, so a bar "at 09:15" can be off by five and a half hours. Sessions have rules: the regular cash session in Indian equities runs from 9:15 to 15:30 India time, with a call auction before it, and the official closing price on the exchange may be a volume-weighted average of the final period rather than the last trade. Check the exchange's current rule before you assume, because a signal built on "the close" is a signal on whichever price the vendor calls the close.
Corporate actions and adjustment
A company can change the number of its shares without changing its value, and the price changes with it. If you do not account for that, a perfectly ordinary day looks like a crash.
Worked example 1: a split. A share closes at ₹1,000 on Friday. The company splits each share of face value ₹10 into five shares of face value ₹2, effective on Monday. Every holder now has five times as many shares and each is worth a fifth as much. On Monday the share opens near ₹200 and closes at ₹200. In the raw data the return from Friday to Monday is 200 ÷ 1,000 − 1 = −80 per cent. No one lost anything: an investor with 10 shares worth ₹10,000 now holds 50 worth ₹10,000. A backtest that reads that −80 per cent as a real return will be wrecked, and a "short the stock before the crash" strategy would look like a stroke of genius.
The fix is to adjust the history: divide every price before the split by 5, so that Friday's close becomes ₹200, and the return from Friday to Monday is 0 per cent. The divisor is the adjustment factor. A bonus issue works the same way. A 1:1 bonus gives each holder one free share for every one held, doubling the share count and halving the price, so earlier prices are multiplied by 0.5. Bonuses and splits are common in India, so an Indian price history without adjustment is almost certainly wrong somewhere.
Worked example 2: a dividend. A share closes at ₹500, and the next day it goes ex-dividend with a dividend of ₹10 per share. On the ex-date it closes at ₹495. The raw price return is 495 ÷ 500 − 1 = −1.0 per cent. But a holder received ₹10, so the total return is (495 + 10) ÷ 500 − 1 = +1.0 per cent. The raw series says the investor lost money, and the investor gained. An adjusted-close series scales earlier prices by (500 − 10) ÷ 500 = 0.98, so that the previous close becomes ₹490 and the return is 495 ÷ 490 − 1 = +1.02 per cent. The small difference from the simple total return of 1.00 per cent is because the adjustment method treats the dividend as if it had been reinvested at the close, and for one day it is negligible. Over many years, ignoring dividends makes a high-yield stock look much worse than it was.
Which prices to use for what
There are two uses of price, and they need different series.
Return calculation and signal building need adjusted prices, so that a split or dividend does not distort the series. Use total-return prices when dividends are material.
Execution simulation needs the prices that really traded: the raw price at the time, with its actual tick size and lot size. If the stock was ₹1,000 on a day, then your order size in shares, your tick of ₹0.05 and any price band at that time all refer to ₹1,000, not to ₹200.
Keep both series and label them. A common bug is to use adjusted prices for the signal and to size orders as if the adjusted price were real.
Try your own case below: enter a price before and after, with a split, a bonus and a dividend, and see how much of the apparent move is real.
Other corporate actions need care. A rights issue gives holders the right to buy new shares at a discount, which lowers the price and needs an adjustment factor based on the terms. A merger or a demerger changes what the share represents: after a demerger the parent's price falls and the new company's price appears with no history. A name or ticker change can break a series in two. Data vendors adjust most actions, but they do not always agree, and an adjustment can be applied late or not at all. The only safe approach is to test: look at the largest daily returns in a series and ask what happened on those days.
Cleaning: bad ticks, gaps and stale prices
Even after adjustment, raw data has errors. Typical ones:
Bad ticks. A price that is obviously wrong: a missing decimal point, a trade printed at ten times the market, a test message. They show up as spikes that reverse at once.
Missing data. A day or a bar with no record, because of a halt, a holiday, a vendor outage or a stock that did not trade.
Stale prices. A price that does not change for days in a thin stock. It is not a missing price, it is a price that is not informative, and it understates volatility.
Duplicates and out-of-order timestamps. The same trade twice, or one that arrives late.
Crossed or locked quotes. A best bid above the best offer, which is impossible in a working market and means a timing error.
Zero or negative volume or price. Usually a placeholder.
Opening and closing auction prints. Real prices that can look odd next to continuous trading.
The key judgement is to separate errors, which you remove, from real extreme moves, which you keep. Removing real crashes makes risk look smaller, which is the same mistake as ignoring fat tails in Lesson 2.2. So use a rule that flags a point for a look, and check a sample by hand.
Worked example 3: a robust spike filter. An ordinary z-score, which uses the mean and standard deviation, is a poor detector, because the bad point itself inflates both. A more robust rule uses the median and the median absolute deviation, the MAD, the median of the distances from the median. Take five prices: 100.2, 100.4, 100.1, 1001.0 and 100.3. The median is 100.3. The distances from the median are 0.1, 0.1, 0.2, 900.7 and 0.0, whose median is the MAD = 0.1. The modified z-score is 0.6745 × (price − median) ÷ MAD. For 100.2 it is −0.67, for 100.4 it is +0.67, for 100.1 it is −1.35 and for 100.3 it is 0. For 1001.0 it is 0.6745 × 900.7 ÷ 0.1 = 6,075. The rule that flags values above about 3.5 catches the spike easily, and nothing else. The typo, a price of 1001.0 where 100.1 was meant, is then fixed or removed, and logged.
When you remove or repair a value, write down what you did. A cleaning step that silently changes the data is another source of untraceable errors, and Lesson 3.2 returns to this point.
Survivorship bias
The most expensive data error is the one in which the data does not contain what is missing. Survivorship bias arises when your dataset includes only the securities that still exist, and leaves out those that failed, were delisted or were taken over. Because the ones that disappeared were, on average, the poor performers, a dataset of survivors looks better than the market it came from.
Worked example 4. Imagine 100 companies at the start of ten years. By the end, 20 have been delisted after poor results, with an average annual return of −10 per cent. The 80 that remain have an average of 14 per cent a year. The true average over all 100 is 0.8 × 14 + 0.2 × (−10) = 11.2 − 2.0 = 9.2 per cent. A researcher who downloads today's list of listed companies, which holds only the 80 survivors, measures 14 per cent, an overstatement of 4.8 percentage points. A strategy tested on that sample has a built-in head start that does not exist in real trading, where you would have owned some of the 20.
The bias appears in many forms. Testing a strategy on today's index members back to 2010 selects stocks that were good enough to be in the index today, so the backtest buys winners by construction. Studies of funds find that databases of surviving funds overstate average performance, because poor funds close or merge, and the effect can be large: see the references at the end of the lesson. The size of the bias grows with the share of failures and with how different their returns are from the survivors'.
Change the numbers below to see when it matters most.
The remedy is survivorship-bias-free data: a database that keeps delisted companies, with their prices up to the day they left, and records the return on delisting. Delisted shares need care too, since a final price of zero, or a long suspension, is a loss that must appear in the return. In a backtest you must also choose the universe as it was on each date, not as it is now: this is called point-in-time membership.
Point-in-time data and look-ahead
A related error is to use information before it was public. Company accounts are published weeks after the period they describe. Quarterly results for the quarter that ended on 31 March are typically announced in the following weeks, often by the middle of May, and the rules allow roughly 45 days. A signal built from March earnings and applied on 1 April uses information nobody had, and the strategy gains from a seven-week head start that cannot be repeated. The same is true of macroeconomic figures, which are published late and then revised: the figure in a database today is usually the final revised one, not the figure traders saw on the day.
The fix is to store, with every number, the date on which it became known, and to use only what was known at each point in the backtest. Data that is "as reported" at each date is called point-in-time, and it is the standard in professional research. We return to look-ahead bias in the backtesting module.
Habits of good data hygiene
1. Never edit the raw data. Keep the files as you received them, with the date and the source, and make every cleaning step a separate, repeatable program that writes a new file.
2. Version your data and note it in every result. If a result changes, you must be able to say whether the model or the data changed.
3. Reconcile two sources. Compare closing prices from two vendors, and investigate any day where they differ by more than a tick.
4. Run sanity checks automatically. Flag any daily return beyond ±20 per cent, any bar with zero volume, any gap longer than the holiday calendar explains, any non-monotonic timestamp.
5. Check the biggest moves by hand. Sort the daily returns and look at the top and bottom ten. Each should have a story.
6. Keep a record of the universe at each date, including delistings.
7. Document the quirks of each vendor, such as how it adjusts, when it stamps bars and what it does on holidays.
The code below applies a split adjustment, flags a bad tick with the median-based rule, and reproduces the survivorship arithmetic.
python
import numpy as np
import pandas as pd
1. split adjustment: a 1-for-5 split takes effect on day 5
raw = pd.Series([1000, 1010, 1005, 1012, 200, 202, 199, 204.0], index=range(1, 9))
adjusted = raw.copy()
adjusted.loc[:4] = raw.loc[:4] / 5
print("raw returns (%) ", (raw.pctchange() 100).round(1).tolist())
print("adjusted returns (%)", (adjusted.pctchange() 100).round(1).tolist())
2. dividend: close 500, ex-date close 495, dividend 10
prior, ex, div = 500.0, 495.0, 10.0
print("price return", round(ex / prior - 1, 4), "total return", round((ex + div) / prior - 1, 4),
"adjusted-close return", round(ex / (prior - div) - 1, 4))
3. robust spike detector: modified z-score from the median and the MAD
prices = np.array([100.2, 100.4, 100.1, 1001.0, 100.3])
med = np.median(prices)
mad = np.median(np.abs(prices - med))
modifiedz = 0.6745 (prices - med) / mad
print("modified z-scores", modifiedz.round(2), "flagged", np.abs(modifiedz) > 3.5)
4. survivorship bias
sharedead, survivors, dead = 0.2, 14.0, -10.0
universe = (1 - sharedead) survivors + sharedead dead
print("whole universe", universe, "bias from survivors only", round(survivors - universe, 1))
5. a sanity check you can run on any return series
def flagreturns(returns, limit=0.20):
return returns[returns.abs() > limit]
print(flagreturns(raw.pctchange().dropna()))
Common mistakes
Using unadjusted prices for returns. A split becomes a crash, and a bonus becomes a collapse.
Using adjusted prices for order sizes and tick rules. The real price at the time is what the market used.
Ignoring dividends. A high-yield stock looks poor in price returns.
Cleaning away real events. Deleting the crash day because it looks like an error makes the strategy look safer than it is.
Using a plain z-score to find outliers. The outlier inflates the standard deviation and hides itself. Use the median.
Downloading today's index list and backtesting it back in time. You buy winners by construction.
Dropping delisted stocks, or recording no return for the last day.
Using a number before it was published. Earnings and revised macro data are late.
Editing the raw file. Nobody can later tell what was changed.
Trusting one vendor. Compare two, and read how each adjusts.
Key takeaways
Data comes as ticks, bars, order books, reference data, corporate actions, fundamentals and alternative data. Each has its own errors, and bars have traps with time zones, sessions and the meaning of "close".
A split makes a raw series show a fake fall: 1,012 to 200 is −80 per cent for a 1-for-5 split. Adjust earlier prices by the factor, 1/5 here, and a bonus of 1:1 uses a factor of 0.5.
A dividend of ₹10 on a ₹500 share turns a raw −1.0 per cent into a total return of +1.0 per cent. Use adjusted prices for returns and raw prices for execution.
A robust spike filter uses the median and the MAD. A bad tick of 1001.0 among values near 100.3 has a modified z-score of 6,075, and the normal values stay below 1.4.
Survivorship bias arises when delisted securities are missing. With 20 of 100 companies delisted at −10 per cent and survivors at 14 per cent, the survivors overstate the whole universe's 9.2 per cent by 4.8 points.
Use point-in-time data: the universe as it was on each date, and each number stamped with the date it became known, so that late earnings and revised figures do not leak the future.
Keep raw data untouched, version it, reconcile two sources, run automatic checks and look at the largest moves by hand.
Check yourself
1. A share trades at ₹2,400, and the company announces a 1:1 bonus. What should the price be after the bonus, roughly, and what factor adjusts earlier prices?
About ₹1,200, since the number of shares doubles. Earlier prices are multiplied by 0.5.
2. A series has these closing prices: 50, 51, 5.2, 52, 51. Is the third value a bad tick, a split or a real crash?
It is almost certainly a bad tick: 5.2 is probably 52 with a missing digit. The price reverses at once, and a split or a crash would not reverse the next day. Check the source, repair it and note the repair.
3. Out of 50 funds ten years ago, 10 have closed, with an average yearly return of −4 per cent. The 40 survivors averaged 9 per cent. What is the true average and the bias?
True average = 0.8 × 9 + 0.2 × (−4) = 7.2 − 0.8 = 6.4 per cent. A survivors-only study says 9 per cent, a bias of 2.6 points.
4. A researcher backtests a strategy on the 2026 list of Nifty 50 companies, from 2012. Name two problems.
Survivorship bias, because only companies that were good enough to be in the index in 2026 appear, and look-ahead, because the researcher is using membership information that was not known in 2012. The list should be the one in force on each date.
5. A company's quarterly results for the quarter ending 30 June are announced on 10 August. A backtest uses them from 1 July. What is wrong?
The backtest uses information about 40 days before it was public, which is look-ahead bias. Results should be used from the announcement date, with a further lag for the time it takes to trade.
Try this
Set aside forty-five minutes. You need daily data for a few Indian shares from a vendor or from the exchange's website.
1. Choose three companies that had a split, a bonus or a large dividend in the last ten years. Download both raw and adjusted closing prices if your source gives them.
2. Compute daily returns from the raw series and sort them. Find the largest falls and rises, and look for the corporate actions behind them.
3. Compare the adjusted series with your own adjustment, using the simulator or the code. Check that the two agree on the action days.
4. Apply the median-and-MAD filter to the returns in a rolling window of 60 days. List the flagged days and check each against news.
5. Take today's list of constituents of an index and the list from five years ago, if you can find it. Count how many names differ, and say what a backtest that used today's list would miss.
6. Use the survivorship simulator with your own estimates of the share that left and their returns, and write down the size of the bias.
7. Write a short data note for one dataset: the source, the date, the adjustments applied, the errors found and how you handled each.
A good result is a data note that someone else could follow to rebuild your clean file, with a table of the corporate actions found and a sentence on how survivorship and look-ahead were avoided or could not be. The next lesson puts clean data to work: how to build a research pipeline that runs the same way every time.
Read more
Stephen Brown, William Goetzmann, Roger Ibbotson and Stephen Ross wrote "Survivorship Bias in Performance Studies" in the Review of Financial Studies in 1992, and Edwin Elton, Martin Gruber and Christopher Blake studied it for mutual funds in "Survivorship Bias and Mutual Fund Performance" in 1996. Ernest Chan's Quantitative Trading has a practical chapter on data problems and backtesting, and Marcos López de Prado's Advances in Financial Machine Learning discusses data structures and point-in-time handling. For adjustment conventions, read your vendor's documentation on how it adjusts for splits, bonuses and dividends before you rely on its numbers.
Tags: Quant Trader Market data Corporate actions Survivorship bias Data cleaning