Lesson 2.3: Regression, correlation and cointegration
Learn correlation, beta and linear regression, why regressing prices on prices misleads, and how cointegration and a spread's half-life underpin pairs trading.
By the end of this lesson you will be able to measure how two return series move together with covariance and correlation, estimate a beta by regression and use it to hedge a position, explain why a regression of one price series on another can look convincing when the two have nothing to do with each other, and describe cointegration and the half-life of a spread, the ideas behind pairs trading. You will show that a stock with a beta of 1.43 to an index needs a hedge of ₹14.3 lakh of index for every ₹10 lakh of stock, that about 87 per cent of regressions between two unrelated random walks look statistically significant, and that a spread that closes 10 per cent of its gap a day has a half-life of 6.6 days. You should have read Lessons 2.1 and 2.2, whose standard deviation and fat tails we use here. The mathematics is averages, products, a slope and a logarithm. This course is education about how markets and research work, not investment advice.
So far we have looked at one series at a time: the returns of one strategy, one share, one index. Most of trading, though, concerns relationships. How much does this stock move when the market moves? Are these two banks really driven by the same forces? If one is cheap relative to the other, will they come back together? The tools in this lesson answer those questions, and they also hide traps. The most important trap is that two unrelated series can show a powerful-looking relationship, and an honest quant learns to recognise it.
Covariance and correlation
The covariance of two series X and Y measures whether they tend to be above their averages at the same time:
Cov(X, Y) = the average of (X − mean of X) × (Y − mean of Y)
If both tend to be high together, and low together, the products are mostly positive and the covariance is positive. If one is high when the other is low, it is negative. The size depends on the units of the two series, which makes it hard to read, so we divide by both standard deviations to get the correlation:
ρ = Cov(X, Y) ÷ (σₓ × σᵧ)
Correlation always lies between −1 and +1. A value of +1 means the two move in perfect lockstep, 0 means no linear relationship and −1 means they move exactly opposite.
Worked example 1. Here are ten days of returns, in per cent, for an index and for one stock: the index 0.8, −0.5, 1.2, −1.0, 0.3, 0.6, −0.7, 1.5, −0.2, 0.4, and the stock 1.1, −0.4, 1.9, −1.6, 0.2, 1.0, −0.9, 2.1, −0.1, 0.9. They are invented numbers, built to look like a stock that moves with the market but more strongly. The index has a mean of 0.24 and a standard deviation of 0.826. The stock has a mean of 0.42 and a standard deviation of 1.195. Their covariance is 0.975, so the correlation is 0.975 ÷ (0.826 × 1.195) = 0.987. The two move almost in lockstep.
What correlation does not tell you
Correlation is a useful summary and a poor guide if taken alone.
It only captures a straight-line relationship. A relationship such as Y = X² has strong dependence but, for symmetric X, correlation near zero.
It is sensitive to outliers. One extreme day can move a correlation from 0.2 to 0.6 in a short sample.
It is not stable. Correlations measured in calm periods often rise sharply in a crisis, when everything falls together, so diversification fails exactly when it is needed.
It does not give size. A correlation of 0.9 says the two move together, not that one moves by the same amount as the other. For that we need a regression.
It is not causation, and between two trending series it can be an illusion, as the next sections show.
Regression and beta
A linear regression fits a straight line through the data to predict one variable from another. For a stock's return rₛ and the market's return rₘ, the market model is
rₛ = α + β rₘ + ε
where α, "alpha", is the average return not explained by the market, β, "beta", is the amount the stock moves for each unit that the market moves, and ε is the leftover noise. The ordinary least squares estimate chooses the line that minimises the sum of squared errors, and has simple formulas:
β = Cov(rₘ, rₛ) ÷ Var(rₘ), and α = mean of rₛ − β × mean of rₘ.
Worked example 2. From the ten days above, the variance of the index is 0.826² = 0.683, so β = 0.975 ÷ 0.683 = 1.43. The intercept is α = 0.42 − 1.43 × 0.24 = 0.08 per cent a day. The stock moves about 1.43 per cent for each 1 per cent move in the index. The R-squared, the share of the stock's variance explained by the index, equals the square of the correlation: 0.987² = 0.974, so 97 per cent. The standard error of the beta is 0.082, so the t-ratio is 1.43 ÷ 0.082 = 17.4. That is overwhelmingly significant, as one would expect from numbers constructed to be clean. Real stock betas are noisier: an R-squared of 0.3 to 0.6 is more typical for a single stock against an index.
Using beta to hedge
The beta tells you how to neutralise the market exposure of a position. If you own ₹10 lakh of the stock, a market fall of 1 per cent costs you about ₹14,300, because the stock falls 1.43 per cent. To cancel that, short an amount of index equal to beta times the position, ₹14.3 lakh, perhaps through index futures. The hedge removes the part of the risk that comes from the market, and what is left is the stock's own noise, the term ε.
How much risk is left? The stock's daily volatility is 1.195 per cent, or about ₹11,950 on ₹10 lakh. After the hedge, the leftover volatility is the residual standard deviation, about 0.19 to 0.20 per cent a day, or roughly ₹2,000. The hedge cut the daily risk by more than 80 per cent. It cannot be perfect: with a correlation of 0.987 you keep a residual of σ × √(1 − ρ²), and the lower the correlation, the less the hedge helps.
Change the correlation and the volatilities below to see what the hedge needs, and how little a low correlation buys you.
The trap: regressing prices on prices
Everything so far used returns, which behave like stable noise around a fixed mean. A tempting shortcut is to regress a price on another price, or an index level on another. Prices trend and wander, and they are not stable in this sense. In 1974 Clive Granger and Paul Newbold showed that regressing one random walk on another, independent of it, produces results that look highly significant, a problem called spurious regression.
Here is how bad it is. We simulated 2,000 pairs of completely independent random walks, each 250 days long, with each step a fresh draw from a normal distribution, so that by construction the two have nothing to do with each other. For each pair we regressed one on the other. The average R-squared was 0.25, and in 17 per cent of the pairs it was above 0.5. A t-ratio above 2 in absolute value, which at a 5 per cent test should appear in 5 per cent of cases by chance, appeared in 87 per cent of the pairs. Anyone who ran this regression and trusted the usual statistics would conclude, nearly nine times in ten, that two unrelated series were related. The result will vary a little with the random draws, but the message does not.
Two lessons follow. First, do not regress trending series on each other and read the usual t-statistics. Second, either work in returns, which are stable, as in the market model, or use the one case in which a relationship between price levels is real and testable: cointegration.
Cointegration: when two wandering series stay together
A random walk wanders without a fixed level, and so does a stock price. Two such series are usually unrelated. Sometimes, though, they are tied together by an economic link, such as two firms in the same industry or a stock and its futures, so that although each wanders, the gap between them cannot wander far. Two series X and Y are cointegrated if some combination
spread = Y − β X
is stable, that is, it fluctuates around a fixed level and keeps returning to it, even though X and Y individually do not.
The standard picture, from Michael Murray's 1994 article "A Drunk and Her Dog", is a drunk wandering at random with her dog on a lead, or off it, also wandering. Each path alone is a random walk. But the dog keeps coming back to its owner, so the distance between them stays bounded. They are cointegrated.
Worked example 3. We simulated an illustrative pair for 60 days. X is a random walk starting at 100, and Y is built as 20 + 1.5 X plus a spread that wanders around zero but pulls back each day by 10 per cent of its gap. Sampled every six days, X runs 100.0, 101.9, 102.8, 103.3, 99.4, 96.3, 96.7, 98.4, 94.6 and 98.0, and Y runs 170.1, 172.2, 175.8, 177.8, 170.6, 165.5, 167.6, 168.3, 161.9 and 169.3. Neither series is stable, and each has moved by several points. But Y minus 1.5 times X, minus 20, stays within a band of about plus or minus three. A trader who knows the hedge ratio of 1.5 can buy one unit of Y and sell 1.5 units of X, and the profit depends only on the spread, not on where the two series go.
Testing for cointegration
The standard recipe, due to Robert Engle and Clive Granger, is the Engle–Granger two-step test:
1. Regress Y on X using price levels, to get the hedge ratio β and the residual, the spread.
2. Test whether the spread is stable, with a test for a unit root such as the Dickey–Fuller test, covered in Lesson 2.4. The usual critical values do not apply here, because the spread was built by fitting, so special tables are used. For a pair of series at a 5 per cent level the cut-off is about −3.3, more demanding than the −2.9 of a plain test.
If the spread passes, the pair is a candidate for pairs trading, a strategy that bets on the spread returning to its normal level: when the spread is unusually wide, buy the cheap leg and sell the expensive one, and close when it narrows.
Trading a spread: the half-life
How quickly does the spread revert? A simple model says the change in the spread each day is proportional to how far it is from its mean:
Δsₜ = b × (sₜ₋₁ − mean) + noise
where b is negative. A b of −0.10 means the spread closes 10 per cent of its gap each day. The half-life, the time for the gap to halve, follows from the fact that after k days the gap is (1 + b)ᵏ of the start:
half-life = −ln 2 ÷ ln(1 + b)
For b = −0.10 it is 0.693 ÷ 0.1054 = 6.6 days. For b = −0.02 it is 34 days, for b = −0.05 it is 13.5 days and for b = −0.20 it is 3.1 days.
A warning about estimating b: it is measured with a large error in a short sample. In the simulated pair above the true b is −0.10 and the true half-life 6.6 days, but a fit on the 60 days gives a hedge ratio of 1.63 instead of 1.5, a b of −0.38 and a half-life of only 1.5 days. On 1,000 days of the same process, the fit gives a hedge ratio of 1.56, a b of −0.10 and a half-life of 6.5 days. A fast reversion measured on two months of data may be noise, so use a long window and check how much the estimate changes when you move it.
The half-life tells you what kind of strategy this is. If it is very short, a day or two, the profit per trade is small and costs dominate. If it is very long, months, your capital is tied up and the relationship may break before the spread closes. Practitioners often look for half-lives of a few days to a few weeks. And the z-score, how many standard deviations the spread is from its mean, tells you when to act: a common rule opens a trade at ±2 and closes at 0, though the right levels are chosen by testing, with costs.
Use the simulator below. It starts the spread two units above its mean, and shows how the gap closes day by day for the speed you choose.
Why pairs trading is harder than it looks
The hedge ratio drifts. A β estimated on last year's data may not hold next month, so it is re-estimated, and each re-estimate changes the trade.
Relationships break. A merger, a regulatory change or a business shock can cut the link for good, and the spread never returns. The loss on a broken pair can wipe out many small wins.
Searching many pairs invites false positives. Test thousands of pairs and some will look cointegrated by luck. Lesson 3.4 shows how to account for that.
Costs matter. Two legs mean two sets of spreads and fees, as Lesson 1.4 showed, and the profit per trade can be small.
Shorting. In Indian equity markets, holding a short position in a share for more than a day is usually done through single-stock futures, which are available for only a limited list of shares. That narrows the choice of pairs.
Leg risk. If one side fills and the other does not, you hold an unhedged position.
The code below computes the correlation, beta and R-squared of the worked example, repeats the spurious regression simulation, and estimates the half-life of a simulated spread.
python
import numpy as np
from scipy import stats
idx = np.array([0.8, -0.5, 1.2, -1.0, 0.3, 0.6, -0.7, 1.5, -0.2, 0.4])
stk = np.array([1.1, -0.4, 1.9, -1.6, 0.2, 1.0, -0.9, 2.1, -0.1, 0.9])
fit = stats.linregress(idx, stk)
print("correlation", round(np.corrcoef(idx, stk)[0, 1], 3), "beta", round(fit.slope, 3),
"alpha", round(fit.intercept, 3), "R2", round(fit.rvalue 2, 3), "t", round(fit.slope / fit.stderr, 1))
print("hedge of 10 lakh needs", round(10 fit.slope, 1), "lakh of index")
spurious regression: independent random walks
rng = np.random.defaultrng(7)
r2, sig = [], []
for in range(2000):
a = np.cumsum(rng.standardnormal(250))
b = np.cumsum(rng.standardnormal(250))
f = stats.linregress(a, b)
r2.append(f.rvalue 2)
sig.append(abs(f.slope / f.stderr) > 2)
print("mean R2", round(float(np.mean(r2)), 2), "share with R2>0.5", round(float(np.mean(np.array(r2) > 0.5)), 2),
"share with |t|>2", round(float(np.mean(sig)), 2))
a cointegrated pair: X a random walk, spread mean-reverting at 10% a day
rng = np.random.defaultrng(11)
x = 100 + np.cumsum(rng.normal(0, 1, 60))
s = np.zeros(60)
for t in range(1, 60):
s[t] = 0.9 s[t - 1] + rng.normal(0, 1)
y = 20 + 1.5 x + s
print("X", np.round(x[::6], 1))
print("Y", np.round(y[::6], 1))
Engle-Granger step 1 and the half-life from the fitted spread, on the 60-day sample and on 1,000 days
def halflife(n):
rng = np.random.defaultrng(11)
x = 100 + np.cumsum(rng.normal(0, 1, n))
s = np.zeros(n)
for t in range(1, n):
s[t] = 0.9 s[t - 1] + rng.normal(0, 1)
y = 20 + 1.5 x + s
step1 = stats.linregress(x, y)
spread = y - (step1.intercept + step1.slope x)
reg = stats.linregress(spread[:-1], np.diff(spread))
return step1.slope, reg.slope, -np.log(2) / np.log(1 + reg.slope), reg.slope / reg.stderr
for n in (60, 1000):
ratio, b, hl, t = halflife(n)
print(n, "days: hedge ratio", round(ratio, 2), "b", round(b, 3), "half-life", round(hl, 1), "t-ratio of b", round(t, 2))
print("true values: hedge ratio 1.5, b -0.1, half-life", round(-np.log(2) / np.log(0.9), 1), "| Engle-Granger 5% cut-off about -3.3")
Common mistakes
Regressing prices on prices. Unrelated random walks look related in about nine cases out of ten, and the t-statistics are meaningless.
Reading correlation as causation or as size. A correlation of 0.9 says nothing about how large the moves are.
Trusting a correlation from a calm period. It can jump in a crisis, so hedges and diversification fail exactly when they are needed.
Using a beta from a short window as a constant. Betas change, so a hedge needs re-checking.
Forgetting the residual. A beta hedge removes market risk, not the stock's own risk.
Assuming every correlated pair is cointegrated. Correlated returns can come from two series whose gap drifts away without limit.
Searching hundreds of pairs and trading the best. Some will pass a cointegration test by luck.
Ignoring the half-life. A reversion too slow, or too fast, makes the pair untradeable after costs.
Key takeaways
Correlation is covariance divided by both standard deviations, between −1 and +1. It measures straight-line co-movement only and is unstable in crises.
The beta from a regression of a stock on the market is Cov ÷ Var. With β = 1.43, ₹10 lakh of stock needs a ₹14.3 lakh index short, and the hedge cut the daily risk from about ₹11,950 to about ₹2,000 in the example.
R-squared equals the correlation squared: the share of variance explained. A hedge cannot remove the residual.
Regressing a trending price on another is spurious: in our simulation 87 per cent of regressions of unrelated random walks looked significant, against the 5 per cent a valid test should give.
Two series are cointegrated if some combination of them, the spread, is stable even though each wanders. The Engle–Granger test regresses, then tests the spread for stability with special critical values.
The half-life of a spread, −ln 2 ÷ ln(1 + b), measures how long reversion takes: 6.6 days when 10 per cent of the gap closes a day, 34 days at 2 per cent.
Pairs trading faces drifting hedge ratios, broken relationships, false discoveries from searching many pairs, costs on two legs and, in India, a limited list of shares that can be shorted.
Check yourself
1. A stock has a daily volatility of 2.0 per cent, the index 1.0 per cent, and the correlation is 0.6. What is the stock's beta?
β = ρ × σₛ ÷ σₘ = 0.6 × 2.0 ÷ 1.0 = 1.2.
2. With that beta, how much index must you short to hedge ₹25 lakh of the stock?
25 × 1.2 = ₹30 lakh of index. The residual risk is σₛ × √(1 − ρ²) = 2.0 × 0.8 = 1.6 per cent a day, so the hedge removes much less than in the lesson's example.
3. A regression of one index's price level on another's gives an R-squared of 0.8 and a t-ratio of 25 over five years of daily data. Should you trust it?
No. Both are trending price levels, so the usual statistics are unreliable, and a spurious regression often looks exactly like this. Regress returns, or test for cointegration.
4. The spread of a pair closes 5 per cent of its gap each day. What is its half-life, and how much of a gap of 4 units remains after 14 days?
Half-life = −ln 2 ÷ ln 0.95 = 13.5 days. After 14 days about 4 × 0.95¹⁴ = 4 × 0.488 = 1.95 units remain, about half.
5. Two stocks are highly correlated in returns. Are they necessarily cointegrated?
No. Correlation of returns measures co-movement in the short run. Cointegration needs a combination of their price levels to be stable, so that the gap does not drift away without limit.
Try this
Set aside forty-five minutes. Pick two shares in the same industry, for example two large private banks, and download at least three years of adjusted daily closing prices.
1. Compute the daily returns of both. Find the correlation, and plot a 60-day rolling correlation. Say how stable it is.
2. Regress the returns of one on the other, find the beta, the R-squared and the standard error of the beta. Work out the hedge for a ₹10 lakh position.
3. Regress one price on the other and note the R-squared and the t-ratio. Then compute the correlation of the changes. Say why the first number looks better than the second.
4. Take the residual of the price regression as the spread. Plot it. Does it look as if it returns to a level?
5. Regress the change in the spread on the previous spread, find b and the half-life, and compute the t-ratio of b. Compare with −3.3.
6. Use the second simulator with your half-life and the current spread to find the z-score and the expected gap after five days.
7. Write a short note: the beta, the hedge, whether the pair looks cointegrated, the half-life, and two reasons the trade could fail.
A good result is a one-page summary with a chart of the spread, an honest statement of whether it looks stable, a half-life and a list of risks. The next lesson goes deeper into the stability that this lesson kept assuming: what stationarity means, how to measure autocorrelation and how to test for a unit root.
Read more
Clive Granger and Paul Newbold's 1974 paper "Spurious Regressions in Econometrics" is the classic warning, and Robert Engle and Clive Granger's 1987 paper "Co-integration and Error Correction" introduced the test. Michael Murray's 1994 article "A Drunk and Her Dog" in The American Statistician is an intuitive account of cointegration. For pairs trading, Evan Gatev, William Goetzmann and K. Geert Rouwenhorst's 2006 paper "Pairs Trading: Performance of a Relative-Value Arbitrage Rule" tests a simple version, and Ganapathy Vidyamurthy's Pairs Trading: Quantitative Methods and Analysis is a practitioner's book.
Tags: Quant Trader Regression Correlation Cointegration Pairs trading