Lesson 4.4: Overfitting, ridge and lasso, and the bias-variance trade-off

Learn why a model that fits the training data perfectly fails on new data, how bias and variance trade off, and how ridge and lasso shrink coefficients.

By the end of this lesson you will be able to recognise overfitting from the gap between training and test error, explain the bias-variance trade-off in plain words, and use ridge and lasso regression to keep a model from chasing noise. You will see training error fall while test error turns upward, compute how a penalty shrinks a slope from 0.87 to 0.58, learn why lasso can set a coefficient to exactly zero, and choose the strength of the penalty with cross-validation. You should have read Lessons 4.1 to 4.3, since regularisation modifies the regression models you have built, and Lesson 2.2 for the idea that a sample is only a sample. The mathematics is sums of squares and one simple formula, worked on the advertising data of earlier lessons.

Every model faces a tension. It must be flexible enough to capture the real pattern, yet not so flexible that it also captures the accidents of the particular data it was trained on. A model that memorises the training set will score brilliantly on it and badly on anything new, and since the point of a model is to predict the unseen, that is failure. Learning to detect and control this tension is probably the most important single skill in modelling. It applies to every model in the rest of the course, from regression to boosted trees to neural networks.

What overfitting looks like

A student who memorises last year's exam paper scores full marks on it and fails the new one. A model can do the same. Overfitting is when a model learns noise as if it were signal: it fits the training rows extremely well because it has bent itself around their random quirks, and so it generalises poorly. Underfitting is the opposite: the model is too simple to capture the real pattern, so it does badly on both the training and the new data.

The way to see the difference is to hold some data back. Fit the model on a training set, and measure its error on a test set it has never seen. Lesson 6.1 covers the mechanics. Here the point is the pattern the two errors make as the model becomes more complex.

Take a regression of sales on advertising spend, fitted with polynomials of increasing degree. A degree-1 model is a straight line, a degree-2 model allows one bend, and a degree-9 model can wiggle through nearly every point. The training error always falls as the degree rises, because a more flexible model can always fit the training points better. The test error falls at first, while the model learns the real shape, and then rises as the model starts to fit noise.

Predict first: looking at that chart, which model would you choose, and what would you conclude about the degree-9 model whose training error is only 0.6? Decide before reading on.

The model to choose is the degree-4 one, where the test error is lowest at 7.0, though degrees 3 and 4 are nearly tied and the simpler degree 3 is the safer pick. The degree-9 model looks excellent on the training data, with an error of 0.6, and is terrible on new data, with an error of 48. If you had judged it only on the data it was fitted to, you would have chosen the worst model of all. This is the central warning of the lesson: never judge a model on the data it was trained on. A large gap between training and test performance is the signature of overfitting. · Underfitting · Good fit · OverfittingTraining error · High · Low · Very low Test error · High · Low · High Gap between them · Small · Small · Large Cause · Model too simple · Right complexity · Model too flexible for the data Typical remedy · More flexible model, better features · Keep it · Simpler model, more data, regularisation ## Bias and variance

There is an elegant way to understand why test error has the shape it does. The expected error of a model on new data splits into three parts.

Bias: error from being wrong on average, because the model is too simple to represent the truth. A straight line fitted to a curve has bias. Variance: error from being sensitive to the particular training sample. A very flexible model changes a lot if a few training rows change. If you refit it on a fresh sample, you get a very different curve. Irreducible noise: randomness in the outcome that no model can predict.

The expected squared error is bias squared plus variance plus noise. As a model gets more flexible, bias falls and variance rises. That is the bias-variance trade-off. A rigid model has high bias and low variance: it is consistently wrong in the same way. A flexible one has low bias and high variance: it is right on average and wildly different from one sample to the next. The best model sits at the complexity where the sum is smallest, and that point depends on how much data you have and how noisy it is. More data lowers variance, so more complex models become affordable as data grows.

With the defaults, total error is lowest at a complexity of about 3.6, where bias squared and variance are nearly equal at about 6.3 each and the total is 21.7. At a complexity of 1 the bias dominates, and at 8 the variance does. Increase the noise slider and the total rises everywhere without moving the sweet spot, because noise is beyond the model's control. Increase the variance scale, as you would with less data, and the best complexity moves left: with little data, simpler is better.

Would more data help? Learning curves

A cheap and revealing diagnostic is the learning curve: fit the same model on increasingly large slices of the training data and record the training and test errors each time. Illustrative results for a fixed model look like this. Training rows · Training error · Test error · Gap50 · 3.0 · 14.0 · 11.0 100 · 4.2 · 10.5 · 6.3 200 · 5.1 · 8.4 · 3.3 400 · 5.6 · 7.3 · 1.7 800 · 5.9 · 6.8 · 0.9 Training error rises with more rows, because there are more points to fit and less room to memorise, and test error falls, so the two converge. The shrinking gap tells you that this model was overfitting at small sizes and that more data cures it. Now read the end of the table: both errors are close to 6, and adding rows barely moves them. If 6 is still too high for your purpose, more data will not help, because the model is limited by bias, and you need a richer model or better features. Two situations call for opposite actions. A big gap that is still closing says collect more data or regularise. A small gap with both errors high says the model is too simple. Plotting the learning curve before you decide how much data to buy, or how elaborate to make the model, can save months.

What to do about overfitting

Several remedies exist, and most work by reducing variance.

Get more data. The most reliable cure, though often the most expensive. Use a simpler model, with fewer predictors, a lower polynomial degree, or a shallower tree. Select features, dropping columns that add noise. Stop training early, for models that learn by iteration. Combine models, as in the ensembles of Module 5. Regularise, which means adding a penalty for complexity to the quantity the model minimises. This is the subject of the rest of the lesson.

Regularisation: a price on large coefficients

Ordinary least squares chooses coefficients to minimise the sum of squared residuals alone. With many predictors, or predictors that overlap, it can use huge positive and negative coefficients that cancel each other to fit noise. Regularisation adds a penalty that grows with the size of the coefficients, so the model must justify each coefficient by an improvement in fit. Two penalties dominate.

Ridge regression minimises the sum of squared residuals plus λ times the sum of the squared coefficients. Lasso regression minimises the sum of squared residuals plus λ times the sum of the absolute values of the coefficients. The number λ, called the regularisation strength, is chosen by you. At λ = 0 you get ordinary regression, and as λ grows the coefficients shrink towards zero. The intercept is not penalised.

For a single standardised predictor the effects have simple forms, which is useful for building intuition. Let b be the ordinary slope and Sxx the sum of squared deviations of the predictor. The ridge slope is

ridge slope = b × Sxx ÷ (Sxx + λ)

and the lasso slope is the ordinary slope reduced by a fixed amount, but never below zero:

lasso slope = max(b − λ ÷ (2 × Sxx), 0)

In the advertising data of Lesson 4.1, b = 0.87 and Sxx = 1,000. With λ = 500, ridge gives 0.87 × 1,000 ÷ 1,500 = 0.58, a shrinkage of one third. With λ = 1,000, ridge gives 0.435. Lasso at λ = 500 gives 0.87 − 0.25 = 0.62, at λ = 1,000 gives 0.37, and at λ = 1,740 or above gives exactly zero.

The two curves show the difference of character. Ridge shrinks every coefficient smoothly, shrinks the large ones the most in absolute terms, and keeps all of them in the model. Lasso, by subtracting a fixed amount, pushes small coefficients to exactly zero, so it performs feature selection as it fits: the predictors it keeps are those that earn their place.

With several predictors the contrast is visible in the coefficients. Suppose ordinary regression gives coefficients of 2.0, 1.5, 0.8, 0.3 and −0.2 for five standardised predictors. Illustrative ridge and lasso fits at the same moderate penalty might give the following. Predictor · Ordinary · Ridge · Lasso1 · 2.0 · 1.6 · 1.4 2 · 1.5 · 1.2 · 0.9 3 · 0.8 · 0.65 · 0.2 4 · 0.3 · 0.25 · 0 5 · −0.2 · −0.17 · 0 Ridge has shrunk everything by about 15 to 20 per cent. Lasso has shrunk the large coefficients a bit and eliminated the two smallest, leaving a model with three predictors that is easier to explain. Which is better depends on the problem. · Ridge · Lasso · Elastic netPenalty · Sum of squared coefficients · Sum of absolute coefficients · A mix of both Coefficients · Shrunk, never exactly zero · Some exactly zero · Some zero, groups kept together Best when · Many predictors, all with some effect, or overlapping predictors · A few predictors matter and the rest are noise · Overlapping predictors and a wish for selection Handles correlated predictors · Spreads the effect across them · Tends to pick one and drop the rest · Compromise Gives a sparse model · No · Yes · Partly Two practical points are essential. First, standardise the predictors before applying a penalty, because the penalty treats all coefficients alike, and a predictor measured in rupees would otherwise be penalised very differently from one measured in lakh. Second, the penalty is a tuning parameter you must choose carefully, which brings us to cross-validation.

Choosing the penalty strength by cross-validation

If λ is too small, the model overfits. If it is too large, the model underfits, with every coefficient squashed towards zero. The sweet spot has to be found from data. The standard method is k-fold cross-validation: split the training data into k parts, say five. Train on four parts and measure the error on the fifth, then rotate so that each part is held out once. Average the five errors. Repeat for a grid of values of λ and pick the one with the lowest average error.

An illustrative result for a ridge model might look like this: an average error of 9.8 at λ = 0.01, 8.9 at 0.1, 8.2 at 1, 8.9 at 10 and 11.5 at 100. The error falls and then rises, the same U shape as before, and the best choice is λ = 1. A common refinement is the "one standard error rule", which picks the largest λ whose error is within one standard error of the best, favouring the simpler model when the data cannot tell them apart.

The test set stays untouched during this selection. It is used once at the end, to report how the final model is expected to perform. If you tune on the test set, you overfit to it, and its error becomes an optimistic story, a mistake that Lesson 6.4 examines closely.

A case where regularisation wins

Consider a problem with 40 predictors and only 100 rows, a common situation in credit and marketing data. Ordinary regression has plenty of freedom to fit noise. In an illustrative test, a model that simply predicts the training average for everyone has a test error of 12.0. Ordinary least squares, fitting all 40 predictors, does worse, with an error of 14.2, because it has overfit. A ridge fit with the penalty chosen by cross-validation reaches 9.1, and a lasso fit reaches 8.7 by discarding most of the 40 predictors.

The lesson of this table is that a more detailed model is not automatically a better one. When data is limited relative to the number of predictors, a constrained model usually beats an unconstrained one. A rough rule for yes-or-no outcomes is to have at least 10 to 20 examples of the rarer outcome for each predictor you want to estimate freely, and where you cannot meet it, you should regularise.

Regularisation in practice

python from sklearn.linearmodel import Ridge, Lasso, RidgeCV, LassoCV from sklearn.preprocessing import StandardScaler from sklearn.pipeline import makepipeline

ridge = makepipeline(StandardScaler(), RidgeCV(alphas=[0.01, 0.1, 1, 10, 100], cv=5)) ridge.fit(Xtrain, ytrain)

lasso = makepipeline(StandardScaler(), LassoCV(cv=5)) lasso.fit(Xtrain, ytrain)

kept = Xtrain.columns[lasso[-1].coef != 0] # the predictors lasso kept print(ridge.score(Xtest, ytest), lasso.score(Xtest, ytest)) # R-squared on unseen data

In scikit-learn the penalty strength is called alpha. For logistic regression the same idea appears as the argument C, which is the inverse of the strength: a small C means a strong penalty. That is why scikit-learn's logistic regression is regularised by default, as the previous lesson noted. Putting the scaler and the model in a pipeline matters, because it makes sure the scaling is learned from the training rows only, a point that returns in the leakage lesson.

Common mistakes

Judging a model on its training data. Training performance always flatters. Use held-out data. Tuning on the test set. Use cross-validation on the training data, and touch the test set once. Forgetting to standardise. An unscaled penalty punishes predictors unequally. Assuming a more complex model is better. With limited data, a simpler model often generalises better. Reading lasso's selection as truth. Which of two overlapping predictors lasso keeps can change from sample to sample. Penalising the intercept. The intercept is not shrunk. Choosing λ by eye. Use cross-validation, and prefer the simpler model when errors are close. Believing regularisation fixes bad data. It reduces variance, and it cannot repair bias, leakage or a flawed sample.

Key takeaways

Overfitting means fitting noise: training error keeps falling while test error rises. A large gap between the two is its signature. Expected error is bias squared plus variance plus noise. More complexity lowers bias and raises variance, so the best model sits in between, and more data moves the sweet spot towards more complex models. Remedies include more data, simpler models, feature selection, early stopping, ensembles and regularisation. Ridge penalises squared coefficients and shrinks them smoothly. Lasso penalises absolute values and sets some to exactly zero. Elastic net mixes the two. Standardise predictors before penalising, and choose the penalty strength by cross-validation, keeping the test set untouched until the end. With few rows and many predictors, a regularised model usually beats an unconstrained one.

Check yourself

1. A model has a training R² of 0.95 and a test R² of 0.60. What is likely happening? Overfitting. It has learned the training data, including noise, and does not generalise. 2. Which part of the error does a more flexible model reduce, and which does it increase? It reduces bias and increases variance. 3. With ordinary slope 0.87 and Sxx = 1,000, what is the ridge slope at λ = 1,000? 0.87 × 1,000 ÷ 2,000 = 0.435. 4. When would you prefer lasso to ridge? When you believe only a few predictors matter and want the model to drop the rest, which gives a sparser, easier-to-explain model. 5. Why should you not choose λ using the test set? The test set would then influence the model, so its error would be optimistic. Use cross-validation on the training data and keep the test set for a final check.

Try this

Use any regression dataset with at least ten predictors and a few hundred rows. Set aside forty-five minutes.

1. Split the data into training and test sets with a fixed random seed. Fit an ordinary regression and record the training and test errors. 2. Fit polynomial models of degree 1 to 6 on one predictor, or add pure noise columns to the regression. Plot training and test error as complexity grows and find where the test error turns up. 3. Standardise the predictors and fit ridge and lasso with cross-validation to choose λ. Record the test error of each. 4. For lasso, list the predictors with non-zero coefficients. For ridge, compare the coefficient sizes with the ordinary fit. 5. Plot the cross-validated error against λ for ridge, and mark the best value and the largest value within one standard error of it. 6. Use the shrinkage simulator to see how a single slope in your model would shrink at a few values of the penalty. 7. Write a short note: where overfitting began, which method generalised best, which predictors survived lasso, and one thing you would still check.

A good result is a plot of training and test error against complexity, a table of test errors for the three methods, and a sentence saying which model you would deploy and why. With that, Module 4 is complete. You can fit, check and tame regression models. The next module introduces models that capture more complex patterns, beginning with how to prepare features for them.

Tags: Data Science Overfitting Regularisation Ridge and lasso Bias-variance