Lesson 11.3: Portfolio theory: mean-variance, CAPM and factor models
Mean-variance optimisation, CAPM and factor models: how portfolios are built, why estimation error wrecks optimisers, and why equal weights are hard to beat.
By the end of this lesson you will be able to explain what the mean-variance frontier is, compute the minimum-variance and maximum-Sharpe portfolios for a small set of assets, say why optimisers are so sensitive to their inputs, describe the capital asset pricing model (CAPM) and estimate a beta with its uncertainty, and use a factor model to split a portfolio's risk into market and stock-specific parts. You will show that three assets with Sharpe ratios of 0.35 to 0.40 combine to a portfolio with a Sharpe ratio of 0.56, that a one-point change in one expected return moves a weight by nine points, that in a simulated test with 20 assets and 10 years of data the naive equal-weight portfolio (Sharpe 0.285) beat the sample-optimal portfolio (0.070), and that a 20-stock portfolio takes 91 per cent of its variance from the market factor. You should have read Lessons 2.3, 11.1 and 11.2. The mathematics is expected values, covariances and matrix algebra. This course is education about how markets and research work, not investment advice.
The previous two lessons dealt with how much to bet and how much you could lose. This one is about how to combine many bets into a single portfolio. The theory dates from Harry Markowitz in 1952, and it is both the foundation of modern investing and a standing example of a method that works perfectly on paper and badly in practice if it is used naively. Understanding why teaches most of what a quant needs about estimation error.
Diversification, in numbers
Lesson 2.3 introduced correlation. The reason it matters is that the volatility of a portfolio is less than the average volatility of its parts when the parts do not move together. For two assets with weights w₁ and w₂, volatilities σ₁ and σ₂ and correlation ρ, the portfolio variance is
σₚ² = w₁²σ₁² + w₂²σ₂² + 2w₁w₂ρσ₁σ₂
When ρ = 1 the portfolio volatility is just the weighted average. When ρ is below 1 it is smaller, and the lower the correlation, the larger the benefit. This is the only free lunch in finance: the expected return of the portfolio is the weighted average of the parts, but the risk is less.
For many assets, the same formula is written with a covariance matrix: σₚ² = wᵀΣw and expected return μₚ = wᵀμ, as in Lesson 11.1.
The mean-variance frontier
Markowitz's idea is to treat the choice of weights as an optimisation. For every level of risk there is a portfolio with the highest expected return, and together these form the efficient frontier. Two portfolios on it are special:
The minimum-variance portfolio has the lowest volatility of any mix of the assets. Its weights are proportional to Σ⁻¹1, where 1 is a vector of ones, and need only the covariance matrix, not expected returns.
The tangency portfolio has the highest Sharpe ratio. With a risk-free rate rf, its weights are proportional to Σ⁻¹(μ − rf1). Any investor who can lend and borrow at rf holds a mix of cash and the tangency portfolio, whatever their taste for risk. This is the two-fund separation result.
Worked example. Take the three assets of Lesson 11.1: equities (expected return 8 per cent, volatility 20 per cent), bonds (5 and 10) and a diversifier (3 and 5), with correlations of 0.3, 0.1 and 0.2, and a risk-free rate of 1 per cent. Their individual Sharpe ratios are 0.35, 0.40 and 0.40. The optimal portfolios are:
Portfolio · Equities · Bonds · Diversifier · Return · Volatility · Sharpe ratioMinimum variance · 1.6% · 13.3% · 85.1% · 3.34% · 4.77% · 0.49
Tangency (maximum Sharpe) · 11.6% · 25.6% · 62.8% · 4.09% · 5.48% · 0.56
None of the three assets has a Sharpe ratio above 0.40, yet the best mix reaches 0.56, a 40 per cent improvement over the best single asset, purely from imperfect correlation. That is the power of diversification, and it explains why a portfolio is not just a collection of ideas but an engineered object. (The minimum-variance Sharpe ratio is (3.34 − 1) ÷ 4.77 = 0.49.)
Both portfolios are heavy in the diversifier, which has the lowest volatility. That is what an optimiser does: it loves assets that are cheap in risk, and it will happily lever them. This leads to the central practical problem.
The optimiser's weakness: estimation error
Mean-variance optimisation is an error maximiser. It takes the inputs, expected returns and covariances, as if they were known, and tilts the portfolio toward the assets whose inputs look best. But expected returns are measured with huge noise (Lesson 10.1: a 6 per cent premium measured from 10 years of 15 per cent volatility data has a standard error of about 4.7 points). The assets whose estimates happen to be too high get the largest weights, and the portfolio is built on errors.
Sensitivity. In the three-asset example, raise the bond's expected return from 5 to 6 per cent, a change well within the estimation error, and the tangency weight in bonds rises from 25.6 to 34.7 per cent, a nine-point shift. Lower the equity return from 8 to 6 per cent and the equity weight falls from 11.6 to 6.3 per cent, nearly halved. Small changes in the inputs produce large changes in the portfolio, which means that rebalancing on updated estimates generates turnover and cost, and that the weights are mostly noise.
An experiment. To measure this, we built a world with 20 assets in which the truth is known: each stock has a beta to the market between 0.6 and 1.4, a stock-specific volatility of 4 to 8 per cent a month, and a small true edge. The best possible portfolio, which an investor with perfect knowledge could hold, has an annualised Sharpe ratio of 0.335. Each of 400 simulated investors saw 120 months (ten years) of returns, estimated the inputs and built a portfolio. We then measured each portfolio's true Sharpe ratio, computed from the actual parameters. Five approaches:
Approach · Mean realised Sharpe · Median · Worst 5% of trialsEqual weights (1/N) · 0.285 · 0.285 · 0.285
Minimum variance, sample covariance · 0.199 · 0.201 · 0.159
Minimum variance, shrunk covariance · 0.216 · 0.217 · 0.188
Mean-variance, sample inputs · 0.070 · 0.075 · −0.066
Mean-variance, shrunk inputs · 0.105 · 0.132 · −0.096
The equal-weight portfolio, which uses no estimates at all, comes within 0.05 of the theoretical best. The sample-optimised portfolio, using the full machinery, achieves 0.070: about a quarter of the equal-weight result, and in 1 trial in 20 it loses money in risk-adjusted terms. Shrinking the inputs (halfway toward a simple target) helps, but not enough to catch 1/N. The result is not specific to this simulation. Victor DeMiguel, Lorenzo Garlappi and Raman Uppal found in 2009 that, in real data sets, 14 optimised portfolio rules rarely beat the equal-weight rule out of sample unless the estimation window was extremely long.
Why does this happen? Because the number of parameters is large relative to the data. With N assets there are N expected returns and N(N + 1) ÷ 2 covariances: 20 assets need 230 numbers, estimated from 120 observations of each. The covariance matrix inverse amplifies the noise in its smallest eigenvalues, so the optimiser finds "arbitrage" between assets that merely look similar in the sample.
What practitioners do
Shrinkage. Pull the sample estimates toward a simple target: a covariance with constant correlation, or a single-factor model, or expected returns toward the cross-sectional mean. Ledoit and Wolf's estimator picks the shrinkage amount from the data.
Factor-model covariances. Estimate a few factor exposures rather than hundreds of correlations (see below).
Constraints. Limits on position size, turnover, sector exposure and leverage prevent extreme weights. A long-only constraint acts as a form of shrinkage.
Black-Litterman. Start from the returns implied by the market portfolio (what the market's weights would justify if investors were optimising), then tilt them with the manager's explicit views, weighted by confidence. The result is a stable starting point rather than a noisy sample mean.
Robust and resampled methods. Optimise over a range of plausible inputs, not a point estimate.
Simple rules. Equal weight, inverse volatility, risk parity (Lesson 11.1) and hierarchical risk parity avoid inverting a noisy matrix.
Penalise turnover. Include trading costs in the objective (Lesson 4.4), so that small changes in the estimates do not cause trades.
The capital asset pricing model
Suppose all investors held mean-variance efficient portfolios and agreed on the inputs. Then, in equilibrium, everyone would hold the same risky portfolio, the market portfolio, and an asset's expected return would depend on only one thing: how much it moves with the market. This is the CAPM of William Sharpe, John Lintner and Jan Mossin (1960s):
E[Rᵢ] − rf = βᵢ × (E[Rm] − rf)
where βᵢ = Cov(Rᵢ, Rm) ÷ Var(Rm) is the beta of the asset to the market, and E[Rm] − rf is the equity risk premium. The line that plots expected return against beta is the security market line. A stock with beta 1.3, in a market with a risk-free rate of 2 per cent and a premium of 5 per cent, has an expected return of 2% + 1.3 × 5% = 8.5%. Only market risk is rewarded; the part of a stock's risk that is specific to it can be diversified away, so no premium is paid for it.
For a fund manager, the CAPM supplies the vocabulary of performance:
Beta is exposure to the market: what you could get from an index fund.
Alpha is the return in excess of what beta explains: Rᵢ − rf = α + β(Rm − rf) + ε. Alpha is the skill, or luck, part.
A market-neutral strategy has a beta near zero: its returns do not depend on the market's direction, which is the aim of the long/short funds in Lesson 9.2.
Estimating beta. Beta is the slope of a regression of the asset's excess returns on the market's. We simulated 60 months of a stock whose true beta is 1.3 and true alpha is 0, with 6 per cent monthly stock-specific volatility. The regression gave:
Estimate · ValueBeta · 1.10 (true value 1.30)
Standard error of beta · 0.18
95% confidence range for beta · 0.76 to 1.45
Alpha, monthly · −0.18%
t-statistic of alpha · −0.21
R² · 0.40
Five years of monthly data gave a beta range wide enough to include 0.8 and 1.4. An investment decision that depends on a beta of 1.1 versus 1.3 is not supported by this. The alpha is statistically indistinguishable from zero, as it should be, since there is none; but the same procedure on a fund with a true alpha of 0.3 per cent a month and the same noise would also give a t-statistic well below 2 (Lesson 3.4). This is why alpha is hard to prove, and why claims of persistent alpha need years of evidence.
Does the CAPM work? Not completely. The empirical security market line is flatter than the model predicts: low-beta stocks have earned more than the theory says and high-beta stocks less, a finding known as the "low-beta anomaly". Further patterns do not fit one factor: small stocks, cheap stocks and recent winners have earned higher returns than their betas explain. That led to richer models.
Factor models
A factor model says that the return of an asset is explained by a handful of common influences, plus a part specific to the asset:
Rᵢ = αᵢ + βᵢ,₁F₁ + βᵢ,₂F₂ + … + εᵢ
The CAPM is a one-factor model with the market as the factor. Eugene Fama and Kenneth French (1993) added size (small minus big) and value (high minus low book-to-market), and later work added profitability, investment and Mark Carhart's momentum. These are the factors of Lesson 5.3. Commercial risk models, of the type sold by Barra, Axioma and others, use dozens of factors: countries, industries and styles such as value, growth, volatility and size.
The factor structure turns the covariance matrix from N(N + 1) ÷ 2 numbers into something manageable:
Σ = BΣF Bᵀ + D
where B holds the factor exposures, ΣF is the covariance of the factors and D is a diagonal matrix of stock-specific variances. With 20 stocks and 3 factors there are 60 exposures, 6 factor covariances and 20 specific variances, instead of 210 covariances, and the estimates are far less noisy.
Uses of a factor model.
1. Risk decomposition. Split a portfolio's variance into factor and specific parts. A portfolio that looks diversified across 100 names may take 80 per cent of its risk from a single value or momentum exposure.
2. Hedging. Offset unwanted factor exposures with futures or a short basket. A "market-neutral" or "sector-neutral" portfolio sets the relevant exposures to zero.
3. Attribution. Explain a period's return as factor contributions plus alpha.
4. Construction. Use the model's covariance in an optimiser, with constraints on exposures.
How diversification works in a factor model
For a single stock with a beta of 1.0 to a market with annual volatility of 15.6 per cent, and a stock-specific volatility of 20.8 per cent a year, the stock's own volatility is 26 per cent and 64 per cent of its variance is stock-specific. Averaging over n such stocks divides the specific variance by n but leaves the market component unchanged:
Number of stocks · Portfolio volatility · Share of variance that is stock-specific1 · 26.0% · 64.0%
5 · 18.1% · 26.2%
10 · 16.9% · 15.1%
20 · 16.3% · 8.2%
50 · 15.9% · 3.4%
100 · 15.7% · 1.7%
Most of the benefit arrives by the time there are 20 to 30 stocks, and the floor is the market itself: diversification removes specific risk but not factor risk. In the 20-asset world of the experiment, with betas that range from 0.6 to 1.4 and an average of 0.97, an equal-weight portfolio gets 90.7 per cent of its variance from the market factor, and an average single stock has an R² to the market of only 34 per cent. This is the reason for the factor approach to risk: you need to manage the exposures that remain after diversification.
Putting it together
A workable approach to building a portfolio from signals uses these pieces.
1. Estimate the risk model: a factor covariance with shrinkage, updated regularly and with a fast-reacting volatility component (Lessons 11.1 and 11.2).
2. Convert signals to expected returns, scaled by the information coefficient and the volatility of each stock (Lesson 10.1), and shrink them toward zero.
3. Optimise with constraints: a risk target, limits on factor exposures, sector and country neutrality, maximum position and trade sizes, and a turnover and trading-cost penalty.
4. Check the result against a simple benchmark such as equal weights or inverse volatility. If the optimised portfolio is not clearly better after costs, it is probably fitting noise.
5. Attribute performance to factors and alpha every period. A strategy that is mostly factor exposure is a cheaper index product under another name.
The code below reproduces the three-asset portfolios, the beta regression, the factor decomposition and the out-of-sample experiment.
python
import numpy as np
from math import sqrt
1. three-asset minimum-variance and tangency portfolios
mu = np.array([0.08, 0.05, 0.03])
vol = np.array([0.20, 0.10, 0.05])
corr = np.array([[1, 0.3, 0.1], [0.3, 1, 0.2], [0.1, 0.2, 1]])
cov = np.outer(vol, vol) corr
rf = 0.01
inv = np.linalg.inv(cov)
ones = np.ones(3)
wmv = inv @ ones / (ones @ inv @ ones)
def tangency(m):
w = inv @ (m - rf)
return w / w.sum()
wt = tangency(mu)
print("Sharpe of each asset", np.round((mu - rf) / vol, 2))
print("min variance", np.round(wmv, 3), "return", round(wmv @ mu, 4), "vol", round(sqrt(wmv @ cov @ wmv), 4), "Sharpe", round((wmv @ mu - rf) / sqrt(wmv @ cov @ wmv), 2))
print("tangency ", np.round(wt, 3), "return", round(wt @ mu, 4), "vol", round(sqrt(wt @ cov @ wt), 4), "Sharpe", round((wt @ mu - rf) / sqrt(wt @ cov @ wt), 3))
print("bonds +1 point:", np.round(tangency(mu + np.array([0, 0.01, 0])), 3), "| equities -2 points:", np.round(tangency(mu - np.array([0.02, 0, 0])), 3))
2. estimating a beta from 60 months of simulated data
rng = np.random.defaultrng(3)
T = 60
mkt = rng.normal(0.007, 0.045, T)
stock = 1.3 mkt + rng.normal(0, 0.06, T)
X = np.columnstack([np.ones(T), mkt])
coef = np.linalg.lstsq(X, stock, rcond=None)[0]
resid = stock - X @ coef
se = np.sqrt(np.diag(resid @ resid / (T - 2) np.linalg.inv(X.T @ X)))
r2 = 1 - resid @ resid / ((stock - stock.mean()) @ (stock - stock.mean()))
print(f"alpha {coef[0]:.4f} (t {coef[0] / se[0]:.2f}) | beta {coef[1]:.2f} se {se[1]:.2f} range {coef[1] - 1.96 se[1]:.2f} to {coef[1] + 1.96 se[1]:.2f} | R2 {r2:.2f}")
3. a world with 20 assets and a known factor structure
rng = np.random.defaultrng(1)
N, T, trials = 20, 120, 400
betas = rng.uniform(0.6, 1.4, N)
idio = rng.uniform(0.04, 0.08, N)
mktvol = 0.045
weq = np.ones(N) / N
fvar = (weq @ betas mktvol) 2
svar = (weq 2 idio 2).sum()
print("equal weights: average beta", round(weq @ betas, 2), "| market share of variance", round(fvar / (fvar + svar), 3), "| average single-stock R2", round((betas 2 mktvol 2 / (betas 2 mktvol 2 + idio 2)).mean(), 2))
for n in (1, 5, 10, 20, 50, 100):
f, i = mktvol 2, 0.06 2 / n
print(f"{n:3d} stocks: volatility {sqrt((f + i) 12):.3f}, specific share {i / (f + i):.3f}")
4. out-of-sample test of portfolio rules
covtrue = np.outer(betas, betas) mktvol 2 + np.diag(idio 2)
mutrue = 0.004 + 0.003 (betas - 1) + rng.normal(0, 0.001, N)
chol = np.linalg.cholesky(covtrue)
sharpeof = lambda w: (w @ mutrue) / sqrt(w @ covtrue @ w) sqrt(12)
wbest = np.linalg.solve(covtrue, mutrue)
print("best possible Sharpe", round(sharpeof(wbest), 3))
results = {k: [] for k in ("equal weights", "min-variance sample", "min-variance shrunk", "mean-variance sample", "mean-variance shrunk")}
for in range(trials):
R = mutrue + rng.standardnormal((T, N)) @ chol.T
m, S = R.mean(0), np.cov(R.T)
sd = np.sqrt(np.diag(S))
rbar = ((S / np.outer(sd, sd)).sum() - N) / (N (N - 1))
Sshrunk = 0.5 S + 0.5 np.outer(sd, sd) (rbar np.ones((N, N)) + (1 - rbar) np.eye(N))
mshrunk = 0.5 m + 0.5 m.mean()
results["equal weights"].append(sharpeof(weq))
results["min-variance sample"].append(sharpeof(np.linalg.solve(S, np.ones(N))))
results["min-variance shrunk"].append(sharpeof(np.linalg.solve(Sshrunk, np.ones(N))))
results["mean-variance sample"].append(sharpeof(np.linalg.solve(S, m)))
results["mean-variance shrunk"].append(sharpeof(np.linalg.solve(Sshrunk, mshrunk)))
for name, v in results.items():
v = np.array(v)
print(f"{name:21s} mean {v.mean():.3f} | median {np.median(v):.3f} | worst 5% {np.percentile(v, 5):.3f}")
Common mistakes
Feeding sample means into an optimiser. The result concentrates in the assets with the luckiest estimates.
Believing the optimal weights are stable. A one-point change in a return moved a weight by nine points.
Not comparing with 1/N. If a complex portfolio cannot beat equal weights out of sample and after costs, it is not adding value.
Treating minimum variance as maximum Sharpe. It ignores returns and can load on low-risk assets that earn less.
Reading a regression beta as exact. With five years of monthly data its range is wide.
Calling a high R² diversification. Holding many stocks removes specific risk, not the common factors.
Ignoring factor exposures. A portfolio of 100 names can be a concentrated bet on one style.
Using CAPM alpha without checking other factors. An alpha against the market may be a known factor premium (value or momentum) under another name.
Optimising without costs and constraints. The unconstrained optimum often calls for extreme positions and heavy trading.
Key takeaways
Portfolio volatility is below the average of the parts when correlations are below 1. Three assets with Sharpe ratios of 0.35 to 0.40 combine to 0.56.
The mean-variance frontier gives a minimum-variance portfolio (covariances only) and a tangency portfolio (maximum Sharpe, weights proportional to Σ⁻¹(μ − rf)).
Optimisers maximise estimation error. A one-point change in an expected return moved a weight by nine points, and in a 20-asset test the sample-optimal portfolio reached a Sharpe ratio of 0.070 while equal weights reached 0.285 and the best possible was 0.335.
Remedies: shrinkage, factor-model covariances, constraints, Black-Litterman views, robust methods, simple rules and turnover penalties.
CAPM says only market beta earns a premium: E[R] = rf + β × premium. Beta is estimated with wide error (1.10 with a range of 0.76 to 1.45 from five years), and alpha is hard to detect.
Factor models write returns as exposures to common factors plus a specific part, and the covariance as BΣF Bᵀ + D.
Diversification removes specific risk (64 per cent of a single stock's variance, 8 per cent for 20 stocks) but not factor risk: a 20-stock portfolio took 91 per cent of its variance from the market.
Check yourself
1. Two assets each have a 10 per cent volatility and a correlation of 0.5. What is the volatility of a 50/50 mix?
Variance = 0.25 × 100 + 0.25 × 100 + 2 × 0.25 × 0.5 × 100 = 25 + 25 + 25 = 75, so the volatility is 8.66 per cent.
2. A stock has a beta of 0.8, the risk-free rate is 3 per cent and the equity risk premium is 5 per cent. What does the CAPM say its expected return is?
3% + 0.8 × 5% = 7.0 per cent.
3. A fund returns 1 per cent a month above the risk-free rate with a beta of 0.5 to a market that returned 1.2 per cent a month in excess of the risk-free rate. What is its alpha?
1.0 − 0.5 × 1.2 = 0.4 per cent a month.
4. Why can a portfolio of 200 stocks still be very risky?
Diversification removes stock-specific risk but not market or factor risk. If the stocks share a beta above 1 or a style tilt, the portfolio still carries that risk.
5. A backtest shows a mean-variance portfolio built on 5 years of monthly data with 50 assets beating equal weights by a wide margin in sample. What is your concern?
With 50 assets, there are 1,275 covariances and 50 means to estimate from 60 observations. The in-sample result is almost certainly overfit, and the out-of-sample test is what counts.
Try this
Set aside ninety minutes. You need Python and the code in this lesson.
1. Run the code and check the tables.
2. Change the number of months in the experiment from 120 to 600. Does mean-variance catch up with equal weights?
3. Change the number of assets from 20 to 5. What happens to the gap?
4. Change the shrinkage weight from 0.5 to 0.8. How do the shrunk results move?
5. Pick five exchange-traded funds. Estimate their covariance from five years of data, compute the minimum-variance portfolio, and change the window to three years. How much do the weights move?
6. Regress a stock's monthly excess returns on the market's and on a second factor such as size or value. Does the alpha stay significant?
A good result is a clear sense of why complicated optimisers lose to simple rules when data are short, a risk model with a handful of factors and a portfolio test that includes costs. The last lesson in this module asks what happens to a portfolio when the world does not look like the sample.
Read more
Harry Markowitz's 1952 paper "Portfolio Selection" in the Journal of Finance founded the subject. William Sharpe's 1964 paper "Capital Asset Prices" presents the CAPM. Eugene Fama and Kenneth French's 1993 paper "Common Risk Factors in the Returns on Stocks and Bonds" introduced the three-factor model. Victor DeMiguel, Lorenzo Garlappi and Raman Uppal's 2009 paper "Optimal Versus Naive Diversification: How Inefficient is the 1/N Portfolio Strategy?" is the evidence behind the lesson's experiment. Olivier Ledoit and Michael Wolf's 2004 paper "Honey, I Shrunk the Sample Covariance Matrix" gives the shrinkage estimator. Fischer Black and Robert Litterman's 1992 paper "Global Portfolio Optimization" explains their approach. For a modern treatment, see Richard Grinold and Ronald Kahn's Active Portfolio Management and Marcos López de Prado's work on hierarchical risk parity.
Tags: Quant Trader Portfolio theory Mean-variance CAPM Factor models