Lesson 4.3: Checking a regression: collinearity, outliers and interactions
Learn to test whether a regression can be trusted: overlapping predictors, fanning residuals, single influential points and effects that depend on each other.
By the end of this lesson you will be able to check a fitted regression for the problems that make its coefficients and intervals unreliable. You will detect overlapping predictors with the variance inflation factor, see why a funnel-shaped residual plot matters and how to repair it, find the single point that can drag a whole line, and add an interaction when one predictor's effect depends on another. You will work through one outlier example by hand, in which a single extra row cuts a slope from 0.87 to 0.18, and one interaction example with festival and ordinary weeks. You should have read Lessons 4.1 and 4.2, since diagnostics check models of both kinds, and Lesson 2.4 for the confounding and outlier ideas. The mathematics is ratios and squares.
A regression always returns an answer. Hand it ten columns of rubbish and it will print ten coefficients, ten p-values and an R², all to three decimal places. The numbers look authoritative whether or not the model deserves it. Diagnostics are the checks that separate a model you can trust from one that merely ran. They are not decoration at the end of a project. A model that fails them can give coefficients with the wrong sign, intervals that are far too narrow, and predictions that collapse on new data.
The assumptions, and what breaks when they fail
Lesson 4.1 listed the assumptions behind a regression's standard errors and intervals. Each failure has a symptom you can look for and a repair you can try.
Problem · What it does · How to see it · Common fixMulticollinearity (overlapping predictors) · Coefficients unstable, huge standard errors · Variance inflation factor, correlation matrix · Drop or combine predictors, use regularisation
Non-constant variance (heteroscedasticity) · Standard errors and intervals wrong · Residuals against fitted values fan out · Log the outcome, weighted fit, robust standard errors
Outliers and influential points · One row drags the fit · Leverage, Cook's distance, plots · Investigate, correct, or model separately
Non-linearity · Systematic misses at the ends · Residuals curve against fitted values · Transform, add squared terms or interactions
Non-independent residuals · Intervals too narrow · Residuals against time or order · Time-series methods, clustered errors
Non-normal residuals · Matters mainly for small samples · Q-Q plot · Transform, larger sample, bootstrap
The first thing to do with any fitted model is to plot the residuals against the fitted values. In a healthy model this plot is a shapeless horizontal band centred on zero. A curve means non-linearity. A funnel means non-constant variance. A few points far from the rest are outliers. Many problems announce themselves in this one picture, which costs almost nothing to draw.
Multicollinearity: when predictors overlap
Suppose a house-price model includes the area in square feet and the area in square metres. They carry exactly the same information, so the model cannot tell which one deserves the credit. Software will refuse or return absurd values. Less extreme cases are far more common: area and number of rooms, income and loan amount, age and years of experience. When predictors are strongly related, the model struggles to separate their effects, and the result is multicollinearity.
The damage shows in the standard errors. Each coefficient's meaning is "the effect of this predictor holding the others fixed", but if two predictors move together, there are few rows in which one changes while the other stays still, so the data has little to say about each one separately. The standard errors balloon, the coefficients become unstable from one sample to the next and may even change sign. Predictions can still be fine, because the combined effect is well determined, which is why the problem goes unnoticed until someone reads the coefficients.
The standard measure is the variance inflation factor, or VIF. For each predictor, regress it on all the other predictors and take that regression's R², written R²j. Then
VIF = 1 ÷ (1 − R²j)
If a predictor is unrelated to the others, R²j is 0 and the VIF is 1, with no inflation. If the others explain half its variation, R²j is 0.5 and the VIF is 2. At R²j of 0.75 the VIF is 4, at 0.90 it is 10, and at 0.95 it is 20. The standard error of that predictor's coefficient is larger by the square root of the VIF, so a VIF of 10 means a standard error about 3.2 times bigger than it would be without the overlap. Common rules of thumb flag a VIF above 5 for attention and above 10 for action, though they are conventions and not laws.
Predict first: two predictors, such as area and number of rooms, have a correlation of 0.95. What is the VIF of each? Work it out before reading on.
With only two predictors, R²j is just the squared correlation between them, 0.95² = 0.9025, so the VIF is 1 ÷ (1 − 0.9025) = 1 ÷ 0.0975 = 10.3. Each coefficient's standard error is about √10.3 = 3.2 times larger than it would be if the two were unrelated. A coefficient that would have been clearly significant may now not be, and the two coefficients will tend to offset each other from sample to sample.
What to do about it
The cure depends on why the predictors overlap. If two columns measure the same thing, keep one. If they measure related things, combine them into one meaningful quantity, such as a ratio of loan to income instead of both separately, or an average of three survey items. If you must keep both, regularisation, which Lesson 4.4 covers, stabilises the coefficients by shrinking them. If the goal is only prediction and not interpretation, multicollinearity matters less, though it still makes the model fragile when the relationship between the predictors changes. A last caution: do not drop a predictor just because it has a high VIF when it is the variable of scientific interest. Think about what you want to learn.
In Python the VIFs come from one function.
python
import pandas as pd
import statsmodels.api as sm
from statsmodels.stats.outliersinfluence import varianceinflationfactor
X = sm.addconstant(df[["area", "rooms", "age", "distance"]])
vifs = pd.Series(
[varianceinflationfactor(X.values, i) for i in range(X.shape[1])],
index=X.columns,
)
print(vifs.drop("const")) # values above about 5 to 10 deserve a look
Non-constant variance
The standard errors of a regression assume the residuals have about the same spread everywhere. Often they do not. Spending on food rises with income, and so does the scatter around the line: households earning ₹20,000 differ in food spend by a few hundred rupees, while households earning ₹1,00,000 differ by thousands. This fanning out is called heteroscedasticity, and in a residual plot it looks like a funnel opening to the right.
The coefficients themselves are still unbiased, but the standard errors are wrong, usually too small where the spread is large, and so confidence intervals and p-values mislead. The usual remedies are practical.
Transform the outcome, most often by taking its logarithm, since many quantities have variation proportional to their size. Modelling log spend turns "the scatter grows with income" into a constant-scatter problem, and the coefficients then read as percentage effects.
Weighted least squares, which gives noisier observations less weight.
Robust standard errors, which correct the standard errors without changing the coefficients. In statsmodels, fit(covtype="HC3") does this, and many analysts use it by default.
A formal check is the Breusch-Pagan test, which tests the hypothesis that the variance is constant. A small p-value signals a problem. But look at the plot too: with a large sample the test can reject for a funnel too mild to matter.
Reading a model of the logarithm
When you model the logarithm of the outcome, the coefficients change meaning, and it pays to know how. Suppose a model for monthly food spending, with the natural log of spend as the outcome, gives a coefficient of 0.04 on household size and 0.012 on income in thousands of rupees. A coefficient on the log scale is roughly a proportional change: each extra household member goes with about a 4 per cent higher food bill, and each extra thousand rupees of income with about 1.2 per cent more. The approximation is good for small coefficients, and the exact change is e^0.04 − 1 = 4.1 per cent. This is natural for money, where a 4 per cent rise means ₹400 for a household spending ₹10,000 and ₹2,000 for one spending ₹50,000, exactly the kind of growing scatter that a plain model would mishandle. When you predict, remember to convert back from the log scale, and that the back-converted prediction is a typical value, not an average, so it is a little low. Many software tools offer a correction, and for business reporting it is often enough to say so.
Outliers and influential points
A single unusual row can rewrite a regression. Three words describe how a point can be unusual, and they are not the same.
An outlier has an outcome far from what the model predicts, a large residual.
A high-leverage point has an unusual value of the predictor, far from the other rows along the horizontal axis, so it has the power to pull the line.
An influential point is one that, if removed, would change the fit a lot. High leverage combined with a large residual makes a point influential.
The advertising example from Lesson 4.1 shows how dramatic this can be. The original five weeks had spend of 10 to 50 and a fitted slope of 0.87 with an R² of 0.97. Now add a sixth week with a spend of 60 and sales of only 20, perhaps because the shop ran out of stock.
With six points the averages become 35 for spend and 38.33 for sales. The sum of products of deviations is 320, and the sum of squared spend deviations is 1,750. The new slope is 320 ÷ 1,750 = 0.18, and the intercept is 38.33 − 0.18 × 35 = 31.9. The sum of squared sales deviations is 1,181.3, so R² = 320² ÷ (1,750 × 1,181.3) = 0.05. One row has cut the slope from 0.87 to 0.18 and the R² from 0.97 to 0.05.
The leverage of the new point can be computed. For a single predictor, the leverage of a row is 1 ÷ n + (x − mean of x)² ÷ the sum of squared deviations. For the sixth week it is 1 ÷ 6 + 25² ÷ 1,750 = 0.167 + 0.357 = 0.52. A typical leverage in a model with one predictor and six rows is about 2 ÷ 6 = 0.33, and a common rule flags a value more than twice that. This point sits well above it. A more complete measure of influence, Cook's distance, combines leverage and residual size into one number for each row, and a common flag is a value above 1, or above 4 divided by n when you want a stricter check.
What should you do with such a point? Never delete it just because it spoils the fit. Investigate. If the stock-out explains it, the week is not comparable and may be removed or flagged with a dummy variable, with the reason written down. If the point is real and typical of what can happen, then the model is wrong, and the straight line may not suit the data. Report the fit with and without the point, so that readers can see how much hangs on one row.
python
influence = fit.getinfluence()
cooksd = influence.cooksdistance[0] # one value per row
leverage = influence.hatmatrixdiag
suspects = df[(cooksd > 4 / len(df)) | (leverage > 2 X.shape[1] / len(df))]
Interactions: when one effect depends on another
So far each predictor has had a single coefficient, an effect that is the same whatever the other predictors are doing. Often that is false. The benefit of advertising may be bigger in a festival week, when people are shopping, than in an ordinary week. The effect of one predictor depends on the level of another, which is an interaction.
To model it, add a column that is the product of the two predictors. Suppose the fitted model is
predicted sales = 20 + 0.6 × spend + 8 × festival + 0.4 × (spend × festival)
where festival is 1 or 0. For an ordinary week the festival terms vanish and the model is 20 + 0.6 × spend, so each extra thousand of spend adds 0.6. For a festival week the model is 20 + 8 + (0.6 + 0.4) × spend = 28 + 1.0 × spend, so each extra thousand adds 1.0. The interaction coefficient, 0.4, is the difference in slopes between the two kinds of week. Without it, the model would force a single slope on both and misstate each.
At a spend of 10, an ordinary week predicts 26 and a festival week 38. At 30, the predictions are 38 and 58. At 50, they are 50 and 78. The gap between the two kinds of week widens as spend grows, which is the interaction made visible.
Interactions are powerful and easy to overuse. Each one adds a coefficient, a risk of overfitting, and a complication in explaining the model. Add one when you have a reason, such as business knowledge or a pattern in the residuals, and check that it improves adjusted R² or predictions on held-out data. When an interaction is in the model, read the main effects carefully: the coefficient 0.6 on spend is now the effect only when festival is 0. A related tool is the squared term, such as spend and spend squared, which lets the slope itself change with the level of spend and so captures curves.
A routine for checking any regression
1. Fit the model and print the summary. Read the coefficients, intervals, R² and adjusted R².
2. Plot residuals against fitted values. Look for curves, funnels and stray points.
3. Compute VIFs. Investigate anything above about 5.
4. Compute leverage and Cook's distance. Look at the most influential rows by hand.
5. Check the residuals for normality and independence, with a Q-Q plot, and with a plot against time if the rows are ordered.
6. Refit after any change, and compare the coefficients with and without the troublesome rows or predictors.
7. Write down what you found and what you did, including anything you chose not to fix.
Common mistakes
Reading coefficients from a model with a high VIF. Their signs and sizes may be unstable.
Deleting outliers to improve the fit. Investigate the reason first, and report the result with and without them.
Trusting a p-value when the residuals fan out. Use robust standard errors or transform the outcome.
Reading R² and ignoring the residual plot. A high R² can hide a curve.
Interpreting a main effect in the presence of an interaction as an average effect. It is the effect when the other predictor is zero.
Adding interactions until the fit improves. Each one adds risk. Justify it and test it on held-out data.
Skipping the checks because the model predicts well. A fragile model fails when the data shifts.
Key takeaways
A regression always returns numbers, so check the assumptions. A residual plot against fitted values reveals curves, funnels and strays in one picture.
Multicollinearity means overlapping predictors. VIF = 1 ÷ (1 − R²j), and standard errors grow by its square root. Drop, combine or penalise.
Non-constant variance leaves coefficients unbiased but makes standard errors wrong. Log the outcome, weight, or use robust standard errors.
Outlier, leverage and influence are different. One high-leverage point with a large residual cut a slope from 0.87 to 0.18, so check leverage and Cook's distance and investigate, never delete by reflex.
An interaction lets one predictor's effect depend on another, and its coefficient is the difference in slopes. Add them for a reason and test them.
Follow a routine and record every change you make.
Check yourself
1. Two predictors are correlated at 0.8. What is the VIF for each?
R²j = 0.64, so VIF = 1 ÷ 0.36 = 2.8. The standard errors are about 1.7 times larger.
2. A residual plot shows a funnel opening to the right. What is the problem and one fix?
Non-constant variance. Take the log of the outcome, or use robust standard errors.
3. What is the difference between a high-leverage point and an outlier?
High leverage means an unusual predictor value that can pull the line. An outlier has a large residual. A point with both is influential.
4. In the festival model, what extra sales does one more thousand of spend bring in a festival week?
The spend coefficient plus the interaction: 0.6 + 0.4 = 1.0 thousand.
5. A VIF check shows that two predictors have VIFs above 12, but you only need predictions. Is the model unusable?
Not necessarily. Predictions can be fine, but the coefficients are unreliable and the model may be fragile if the relationship between the predictors shifts.
Try this
Use the model you fitted in Lesson 4.1 or any regression with at least three predictors. Set aside forty-five minutes.
1. Plot the residuals against the fitted values. Describe the shape in one sentence.
2. Compute the VIF of each predictor. For any above 5, write what the overlap is and one way to deal with it.
3. Find the three rows with the largest Cook's distance. Look at each, and decide whether each is an error, a special case or ordinary.
4. Refit without the most influential row. Compare the coefficients and the R² with the original, as in the advertising example.
5. Pick two predictors you think might interact, add their product, and see whether adjusted R² or a holdout error improves. Describe the effect of one predictor at two levels of the other, using the interaction simulator if it helps.
6. If the residuals fan out, refit with a log outcome or with robust standard errors and compare the intervals.
7. Write a short audit note: what you checked, what you found, what you changed and what you left alone.
A good result is a note that someone could use to decide whether to trust the model, with at least one decision that changed because of a check. The next lesson meets the opposite danger: a model that fits the training data too well, and the tools, ridge and lasso, that keep it honest.
Tags: Data Science Regression diagnostics Multicollinearity Outliers Interactions