Lesson 10.3: Deep learning and reinforcement learning, and their limits
Learn how neural networks and reinforcement learning work, why low signal and short histories limit them in finance, and where they do help.
By the end of this lesson you will be able to describe how a neural network is built and trained, count its parameters, explain why deep learning has transformed images and language but struggles with the noise of returns, read a learning curve to decide whether more data or a bigger model would help, describe reinforcement learning with states, actions and rewards, and say where each technique is worth using in quantitative finance. You will show that a 32-by-16 neural network reaches an out-of-sample R-squared of 46 per cent on data with a strong non-linear signal once it has 32,000 examples, beating a linear model's 25 per cent, but never beats the linear model on data where the best possible R-squared is 0.5 per cent, even with 32,000 examples, and that a tabular reinforcement-learning trader trained on a mean-reverting price earns a test Sharpe ratio of 0.79 against 1.63 for a two-line rule, varying from −0.26 to 1.65 across random seeds. You should have read Lessons 3.4, 4.3 and 10.1. The mathematics is matrices, derivatives and averages. This course is education about how markets and research work, not investment advice.
The newspapers speak as if deep learning could predict markets, and the companies that sell it do not discourage the idea. Lesson 10.1 gave the sober picture for ordinary machine learning: the signal in returns is a small fraction of one per cent of the variance, and more flexible models mostly add noise. This lesson looks at the two most powerful families of the day, deep neural networks and reinforcement learning. Both have remarkable successes elsewhere, and both are useful in finance in specific places. Understanding why they work where they do, and fail where they fail, is the best protection against both hype and dismissal.
What a neural network is
A neural network is a function built by stacking simple pieces. The basic piece is a neuron: it takes a vector of inputs x, multiplies by weights w, adds a bias b and passes the result through a non-linear activation function such as the rectified linear unit, max(0, z):
output = activation( w · x + b )
A layer is a set of neurons that all look at the same inputs. A deep network stacks several layers, the output of one feeding the next. A network with two hidden layers of 32 and 16 neurons, taking 20 inputs and producing one forecast, computes
f(x) = w₃ · ReLU( W₂ · ReLU( W₁ x + b₁ ) + b₂ ) + b₃
The network is trained by choosing the weights to minimise a loss function, such as the mean squared error between its forecasts and the realised values. The standard method is gradient descent: compute the derivative of the loss with respect to every weight, using the chain rule layer by layer (backpropagation), and move the weights a small step against the gradient, repeating over the data many times. Several habits keep training stable: starting values scaled to the layer size, mini-batches of data, a learning rate that sets the step size, weight decay that penalises large weights, dropout that randomly switches off neurons, and early stopping when the error on held-out data stops falling.
Counting parameters
The weights and biases are the parameters, and they are what the network can fit. For the network above, with 20 inputs, 32 and 16 hidden neurons and one output:
first layer: (20 + 1) × 32 = 672
second layer: (32 + 1) × 16 = 528
output layer: (16 + 1) × 1 = 17
The total is 1,217 parameters. A linear model on the same 20 inputs has 21. A theorem of the 1980s, the universal approximation theorem, says that a network with enough neurons can approximate any reasonable function. That is the source of the promise, and also of the danger: a function that can represent anything can also represent the noise.
The families
Multilayer perceptrons (MLPs), as above, for tabular data such as a vector of features per stock.
Convolutional networks (CNNs) for data with a grid structure, such as images, or price charts treated as images.
Recurrent networks and LSTMs for sequences, such as a time series of prices or order-book snapshots.
Transformers, built on an attention mechanism that lets each position in a sequence look at all the others. They underlie the large language models of Lesson 10.4 and are now used for sequences of every kind.
Autoencoders that compress data into a few numbers, used to find factors and to detect anomalies.
Why deep learning works elsewhere
The great successes of deep learning share four features, and finance lacks most of them.
· Image and speech recognition · Return predictionSignal-to-noise · High: a picture of a cat is clearly a cat · Very low: R² of a fraction of one per cent
Data · Millions to billions of labelled examples · A few hundred months, a few thousand stocks, one history
Stationarity · The meaning of a cat does not change · Relationships change with the market regime
Repeated experiments · You can collect more data in the same conditions · You cannot rerun 2008; the sample is what history gave
Cost of an error · A misclassified photo · Real money, and a market that reacts to you
Feedback · The pattern is not changed by being found · Others find it, trade on it, and it decays
## A test: learning curves
A learning curve plots a model's out-of-sample performance against the amount of training data. It tells you whether a model is limited by lack of data (the curve is still rising), by the model's form (the curve has flattened below the ceiling), or by noise.
We built two settings. In both, the features are 10 independent normal variables. The target is a mix of a linear part (two features) and a non-linear part (a product of two features passed through a hyperbolic tangent, and a squared term), in equal variance, plus noise. In the high-signal setting the signal explains 50 per cent of the variance at best. In the finance-like setting it explains 0.5 per cent at best, like the market of Lesson 10.1. We compared a linear model with a 32-by-16 network, tested on 20,000 fresh examples, averaging three repeats.
High signal (ceiling 50 per cent):
Training examples · Linear model R² · Neural network R²500 · 23.3% · 6.7%
2,000 · 24.6% · 22.9%
8,000 · 24.6% · 37.7%
32,000 · 24.9% · 45.9%
Finance-like (ceiling 0.5 per cent):
Training examples · Linear model R² · Neural network R²500 · −1.30% · −10.59%
2,000 · −0.44% · −2.92%
8,000 · +0.03% · −0.51%
32,000 · +0.20% · +0.01%
The two tables tell opposite stories. With a strong signal, the linear model is stuck at about 25 per cent, since it can capture only the linear half of the structure. The network is worse with 500 examples, because it has 1,000 parameters and little to learn from, equal with 2,000, and then far better: 37.7 per cent with 8,000 and 45.9 per cent with 32,000, near the ceiling. More data and more flexibility pay.
With a finance-like signal, the network is never better than the linear model, even at 32,000 examples. It is much worse at small sizes, −10.6 per cent against −1.3 per cent at 500, and it only just reaches zero at 32,000, while the linear model has already reached 0.2 per cent. The network has to learn the noise away before it can use the signal, and the signal is too small for that to pay off. To benefit, it would need far more data than a monthly stock panel can offer.
Count how many parameters your own network has against the data you have.
Where deep learning does help in finance
High-frequency and microstructure data. An order book generates millions of observations a day, the signal-to-noise ratio over seconds is higher than over months, and the relationships are more stable. Deep models of order flow and short-term price moves have shown value, with the caveat that costs and latency decide whether a forecast can be traded (Lessons 4.4 and 5.4).
Text, images and other unstructured data (Lessons 10.2 and 10.4), where pretrained models carry knowledge from vast data sets that no finance data set could supply.
Targets with more signal than returns: volatility, volume, spreads, the probability of default, the time to execute an order. These have strong persistence and structure.
Compressing and denoising: autoencoders for factor structure, anomaly detection for data errors and operational risk.
Combining many forecasts and features with strong regularisation in the way of Lesson 10.1, where the gains over linear models are modest but measurable.
Simulating and generating data for stress testing, with care.
The practical rules: use small networks, with strong regularisation and early stopping; ensemble many of them; use a validation period of the future, not a random split (Lesson 4.3); compare with a linear baseline at every stage; and treat any large improvement as suspect until a leak has been ruled out.
Reinforcement learning
Supervised learning, the setting so far, learns a mapping from inputs to known outputs. Reinforcement learning (RL) learns how to act in an environment so as to maximise a long-run reward. It fits a trading problem naturally: the trader observes the market, chooses a position, receives profit or loss and moves on, and what it does now affects what it can do later.
The framework, a Markov decision process, has five parts:
a state s: what the agent knows, such as the price signals and the current position,
an action a: what it can do, such as buy, sell or hold,
a reward r: the immediate gain, such as the profit minus costs,
a transition to a new state, driven partly by the market,
a discount factor γ between 0 and 1 that weights future rewards.
A policy maps states to actions. The value of a state-action pair, Q(s, a), is the expected total discounted reward from taking action a in state s and following the policy afterwards. Q-learning learns Q from experience with the update
Q(s, a) ← Q(s, a) + α × [ r + γ × max over a′ of Q(s′, a′) − Q(s, a) ]
where α is a learning rate and s′ the next state. The term in brackets is the error between what the agent expected and what it got plus its estimate of what comes next. The agent mostly takes the action with the highest Q, and sometimes a random one, to explore. In deep reinforcement learning a neural network replaces the table of Q values, so that large state spaces can be handled.
Where RL is used in finance
Optimal execution: splitting an order over time to minimise cost, with the state including the time left, the shares remaining and market conditions, in the spirit of Lesson 5.4.
Market making: choosing quotes given the inventory and the order book.
Portfolio allocation and hedging: choosing positions or hedges that account for costs and for how today's trade changes tomorrow's options.
It is attractive where decisions are sequential and costs link today to tomorrow, which is exactly where a one-period prediction model is weakest.
Why RL is harder in finance than in games
RL's great successes are in games and robotics, where an agent can play millions of times against a perfect simulator. In trading:
There is no faithful simulator. A backtest that replays history cannot show how the market would have reacted to your orders. An agent trained on historical replays learns to exploit the replay, and a simulated market built from assumptions teaches the agent the assumptions.
Rewards are noisy and delayed. A profit on one trade says little about whether the decision was good, so learning is slow and unstable.
The environment is non-stationary. What was learned in one regime may be wrong in the next.
There is one history. The agent sees the same sample over and over, so it overfits to its idiosyncrasies.
Exploration costs real money. In a game a bad move costs nothing, in a market it costs capital.
It is easy to hack the reward. An agent maximising backtest profit can find loopholes, such as exploiting bid–ask bounce or unrealistic fills, that the real market does not offer.
A toy trader
To see these problems at work, we built the simplest reinforcement learner and gave it an easy task. The price is a mean-reverting spread, as in Lesson 5.2: each day it moves to 0.97 of its previous value plus a normal shock. The agent sees which of seven bands the spread is in, from far below its mean to far above, and its current position of −1, 0 or +1. It chooses its next position, earns the change in the spread times its position, and pays a cost of 0.05 for every unit of change in position. We trained a tabular Q-learner (a table of 7 × 3 × 3 values, no neural network) on 40,000 days, with 20 different random seeds, and tested the learned policy on fresh 10,000-day paths. As a benchmark we used a simple rule: go long in the two lowest bands, short in the two highest, and hold the position until the spread returns towards the middle.
· Test Sharpe ratio, same dynamics · Test Sharpe ratio, mean reversion weakened (0.995)Reinforcement learner (average of 20 seeds) · 0.79 · 0.38
Spread of the learner's results across seeds · −0.26 to 1.65 (standard deviation 0.56) · standard deviation 0.23
Simple rule · 1.63 · 0.71
The results are humbling.
The learner underperforms a two-line rule. On average it earns about half the rule's Sharpe ratio, and in every one of the 20 seeds it did worse.
It is unstable. With the same data and the same algorithm, changing only the random seed produced test Sharpe ratios from −0.26 to 1.65. A result from one training run means little.
It does not transfer. When the mean reversion weakened, the learner's Sharpe ratio fell to 0.38, as did the rule's to 0.71, with neither adapting.
A more elaborate learner, with a network and more data, would do better on this toy problem. The lesson is not that RL cannot work, but that it must be benchmarked against a simple rule and tested for stability, and that in a problem whose structure you understand, writing the structure down is usually better than making an agent rediscover it.
Guidelines
1. Start with the problem, not the tool. Is the signal strong or weak, and is the data large or small? A weak signal and a small sample favour simple models.
2. Benchmark against baselines at every step, linear models for forecasts and simple rules for policies.
3. Plot a learning curve. If it is still rising, more data will help. If it is flat and low, a bigger model will not.
4. Test for stability. Repeat the training with different seeds and data splits, and report the spread, not the best run.
5. Use the right data for the model. Deep learning needs a lot of data, which means high-frequency data, text, images or pooled data, not a few hundred monthly observations.
6. Keep a clean out-of-sample test, use time-ordered splits, and account for the number of models tried (Lesson 3.4).
7. Check the economics. Include costs, capacity and market impact, which an RL agent trained on a replay never sees.
8. Plan for monitoring and decay. A model you cannot explain is hard to switch off for the right reason (Lesson 12.2).
The code below builds the learning-curve experiment and the toy Q-learning trader, using only numpy.
python
import numpy as np
from math import sqrt
---- a small neural network trained by backpropagation ----
def fitmlp(X, y, hidden=(32, 16), epochs=60, lr=0.01, l2=1e-4, seed=0):
rng = np.random.defaultrng(seed)
scale = y.std()
target = y / scale
sizes = [X.shape[1], hidden, 1]
W = [rng.normal(0, sqrt(2 / sizes[i]), (sizes[i], sizes[i + 1])) for i in range(len(sizes) - 1)]
B = [np.zeros(sizes[i + 1]) for i in range(len(sizes) - 1)]
for in range(epochs):
order = rng.permutation(len(X))
for s in range(0, len(X), 256):
b = order[s:s + 256]
acts = [X[b]]
for i in range(len(W) - 1):
acts.append(np.maximum(acts[-1] @ W[i] + B[i], 0)) # ReLU layers
out = (acts[-1] @ W[-1] + B[-1]).ravel()
g = (out - target[b])[:, None] / len(b)
for i in range(len(W) - 1, -1, -1):
gW, gB = acts[i].T @ g + l2 W[i], g.sum(axis=0)
if i > 0:
g = (g @ W[i].T) (acts[i] > 0)
W[i] -= lr gW; B[i] -= lr gB
def predict(Z):
a = Z
for i in range(len(W) - 1):
a = np.maximum(a @ W[i] + B[i], 0)
return (a @ W[-1] + B[-1]).ravel() scale
return predict
def fitlinear(X, y):
Xb = np.c[np.ones(len(X)), X]
w = np.linalg.solve(Xb.T @ Xb, Xb.T @ y)
return lambda Z: np.c[np.ones(len(Z)), Z] @ w
def truth(X):
lin = X[:, 0] 0.8 + X[:, 1] -0.6
non = np.tanh(X[:, 2] X[:, 3]) + 0.5 (X[:, 4] 2 - 1)
return 0.7 lin / lin.std() + 0.7 non / non.std()
def learningcurvepoint(signalsd, N, seed, test=20000):
rng = np.random.defaultrng(seed)
Xtr, Xte = rng.standardnormal((N, 10)), rng.standardnormal((test, 10))
scale = signalsd / truth(rng.standardnormal((50000, 10))).std()
ytr = truth(Xtr) scale + rng.normal(0, 1, N)
yte = truth(Xte) scale + rng.normal(0, 1, test)
r2 = lambda f: 1 - ((yte - f(Xte)) 2).sum() / ((yte - yte.mean()) 2).sum()
return 100 r2(fitlinear(Xtr, ytr)), 100 r2(fitmlp(Xtr, ytr, seed=seed))
print("parameters of a 20-32-16-1 network:", (20 + 1) 32 + (32 + 1) 16 + (16 + 1), "| a linear model on 20 inputs:", 21)
for label, sd in (("strong signal (ceiling 50%)", 1.0), ("finance-like signal (ceiling 0.5%)", 0.07)):
print(label)
for N in (500, 2000, 8000, 32000):
pts = np.array([learningcurvepoint(sd, N, s) for s in range(3)])
print(f" N = {N:>5}: linear R2 {pts[:, 0].mean():7.2f}% | neural network R2 {pts[:, 1].mean():7.2f}%")
---- a tabular Q-learning trader on a mean-reverting spread ----
def ou(n, phi, seed):
rng = np.random.defaultrng(seed)
x, e = np.zeros(n), rng.normal(0, 1, n)
for t in range(1, n):
x[t] = phi x[t - 1] + e[t]
return x
SD = 1 / sqrt(1 - 0.97 2)
BANDS = np.array([-2.0, -1.0, -0.4, 0.4, 1.0, 2.0]) SD # seven states
state = lambda v: int(np.searchsorted(BANDS, v))
COST = 0.05
def runpolicy(x, policy):
pos, pnl = 0, []
for t in range(len(x) - 1):
new = policy(state(x[t]), pos)
pnl.append(new (x[t + 1] - x[t]) - COST abs(new - pos))
pos = new
return np.array(pnl)
def sharpe(p):
return p.mean() / p.std() sqrt(252) if p.std() > 0 else 0.0
def simplerule(s, pos):
if s <= 1:
return 1 # far below the mean: buy
if s >= 5:
return -1 # far above the mean: sell
if s == 2 and pos == 1 or s == 4 and pos == -1:
return pos # hold until the spread is back near the middle
return 0
def qlearn(x, seed, eps=0.2, alpha=0.1, gamma=0.95):
rng = np.random.defaultrng(seed)
Q = np.zeros((7, 3, 3)) # state, position (-1, 0, 1), action
actions, pos = [-1, 0, 1], 0
for t in range(len(x) - 1):
s = state(x[t])
a = rng.integers(3) if rng.random() < eps else int(Q[s, pos + 1].argmax())
new = actions[a]
reward = new (x[t + 1] - x[t]) - COST abs(new - pos)
Q[s, pos + 1, a] += alpha (reward + gamma Q[state(x[t + 1]), new + 1].max() - Q[s, pos + 1, a])
pos = new
return lambda s, p: [-1, 0, 1][int(Q[s, p + 1].argmax())]
train = ou(40000, 0.97, 1)
rows = []
for seed in range(20):
policy = qlearn(train, seed)
rows.append([sharpe(runpolicy(ou(10000, 0.97, 100 + seed), policy)), sharpe(runpolicy(ou(10000, 0.995, 200 + seed), policy)),
sharpe(runpolicy(ou(10000, 0.97, 100 + seed), simplerule)), sharpe(runpolicy(ou(10000, 0.995, 200 + seed), simplerule))])
rows = np.array(rows)
print("learner, same dynamics: mean", rows[:, 0].mean().round(2), "sd", rows[:, 0].std().round(2), "range", rows[:, 0].min().round(2), "to", rows[:, 0].max().round(2))
print("rule, same dynamics: mean", rows[:, 2].mean().round(2))
print("weaker mean reversion: learner", rows[:, 1].mean().round(2), "| rule", rows[:, 3].mean().round(2))
print("share of seeds where the learner beats the rule:", (rows[:, 0] > rows[:, 2]).mean())
Common mistakes
Assuming a deeper network means a better forecast. With a weak signal, depth adds variance.
Judging by a single training run. Neural networks and RL agents vary with the seed. Report the spread.
Using a random split on time series. The future leaks into training.
Tuning the architecture on the test set. It becomes training data.
Training an RL agent on a replay and trusting the result. It learns the replay, not the market, and has no model of its own impact.
Ignoring costs in the reward. An agent will find high-turnover tricks that vanish after costs.
Not benchmarking against a simple rule. A tabular learner earned half the Sharpe ratio of a two-line rule.
Using deep learning on a few hundred monthly observations. It needs far more data than that.
Treating a black box as a strength. If you cannot say why it works, you cannot say when it stopped.
Key takeaways
A neural network stacks layers of weighted sums and non-linear activations and is trained by gradient descent with backpropagation. A 20-32-16-1 network has 1,217 parameters, against 21 for a linear model.
Deep learning succeeds where the signal is strong, data is huge and the world is stationary. Returns have a tiny signal, a short single history and changing relationships.
Learning curves show the difference. With a strong signal, a network beat a linear model from about 4,000 examples and reached 45.9 per cent against 24.9 per cent at 32,000. With a signal worth at most 0.5 per cent, it was worse at every size and reached only 0.01 per cent against 0.20 per cent at 32,000.
Deep learning helps with high-frequency data, text and images, targets with more signal than returns, compression and denoising, and in regularised combinations of many forecasts.
Reinforcement learning learns a policy from states, actions and rewards, with Q-learning updating Q ← Q + α[r + γ max Q′ − Q]. It suits sequential decisions with costs, such as execution, but lacks a faithful simulator, has noisy rewards, a non-stationary world and one history.
In a toy mean-reverting market, a tabular Q-learner earned a test Sharpe ratio of 0.79 against 1.63 for a simple rule, ranged from −0.26 to 1.65 across seeds, and fell to 0.38 when the dynamics changed.
Benchmark against baselines, plot learning curves, test stability, keep a clean out-of-sample test and include costs.
Check yourself
1. A network has 50 inputs, a first hidden layer of 64 neurons, a second of 32 and one output. How many parameters does it have?
(50 + 1) × 64 + (64 + 1) × 32 + (32 + 1) = 3,264 + 2,080 + 33 = 5,377.
2. It is trained on 10,000 stock-months of returns. Why is that a worry?
Fewer than two examples per parameter, with a target that is almost all noise. The network will fit the noise, and its out-of-sample R² will probably be negative.
3. A learning curve for a model is flat at an R² of 0.1 per cent from 10,000 to 100,000 examples. What does it suggest?
More data will not help: the model has reached what it can extract, either because the signal is that small or because the model is the wrong form. A different model or feature set is needed.
4. In Q-learning, the current Q value is 0.2, the reward is −0.1, the best next value is 0.3, α = 0.2 and γ = 0.9. What is the new value?
Target = −0.1 + 0.9 × 0.3 = 0.17. Error = 0.17 − 0.2 = −0.03. New Q = 0.2 + 0.2 × (−0.03) = 0.194.
5. An RL agent trained on five years of historical prices earns a Sharpe ratio of 3 in the backtest on the same five years. What should you suspect?
Overfitting to the training history: the agent has seen the data many times and has learned its idiosyncrasies. Test on later data it has not seen, and across seeds.
Try this
Set aside forty-five minutes. You need Python and the code in this lesson, which takes a few minutes to run.
1. Run the code. Check the learning-curve tables and the Q-learning results.
2. Change the signal in the finance-like setting to 0.2, which has a ceiling of about 4 per cent. At what training size does the network overtake the linear model?
3. Add weight decay of 0.01 and early stopping to the network, and repeat the finance-like curve. Does it close the gap?
4. In the Q-learning experiment, use 10 states instead of 7 and train for 100,000 days. Does the learner approach the rule?
5. Train the learner on a spread with a mean-reversion coefficient of 0.97, and test it at 0.9, 0.97 and 1.0. Where does it lose money?
6. Use the Q-learning simulator to compute an update by hand and check it against the code.
7. Write a short note: one problem in your work where deep learning or RL might help, the data you would have, the baseline you would beat and the test that would convince you.
A good result is learning curves for a strong and a weak signal, a Q-learning trader that you have benchmarked against a rule and a stability test across seeds. The last lesson of this module turns to large language models, which bring a different set of promises and hazards.
Read more
Ian Goodfellow, Yoshua Bengio and Aaron Courville's Deep Learning is the standard text. Richard Sutton and Andrew Barto's Reinforcement Learning: An Introduction is the classic on RL, freely available online. Matthew Dixon, Igor Halperin and Paul Bilokon's Machine Learning in Finance: From Theory to Practice covers neural networks and RL for trading. Marcos López de Prado's Advances in Financial Machine Learning explains why standard methods fail on financial data, and Gu, Kelly and Xiu's 2020 paper (Lesson 10.1) compares networks and linear models on stock returns. For the limits, see Campbell Harvey and Yan Liu's work on backtest overfitting and Thomas Fischer and Christopher Krauss's 2018 paper on LSTM networks for market prediction in the European Journal of Operational Research.
Tags: Quant Trader Deep learning Reinforcement learning Neural networks Overfitting