Lesson 4.1: Linear regression, R-squared and adjusted R-squared

Learn how a regression line is fitted, how to read slopes, R-squared and adjusted R-squared, and how to add several predictors without fooling yourself.

By the end of this lesson you will be able to fit a straight line to data by hand, read its intercept and slope in plain words, judge how well it fits with R-squared, and explain why adjusted R-squared is the honest version when you add more predictors. You will fit a line to five points, compute every number that a software summary prints, extend the idea to several predictors and dummy variables, and see how a model with extra junk columns can look better and be worse. You should have read Lesson 2.4, since regression is the correlation of that lesson turned into a prediction tool, and Lesson 2.3 for standard errors and t statistics. The mathematics is sums, products and one square root, and the Python is a few lines.

Regression is the workhorse of data science. A lender predicts the loss on a loan, a retailer predicts demand, an economist estimates the effect of a policy. Newer methods such as boosted trees often predict better, but regression is still the first model to try, because it is simple, fast, and above all explainable: each input has a coefficient that says what it contributes. A team that cannot explain a model to a regulator or a manager often cannot use it, and regression makes explanation easy.

The idea: the best straight line

Lesson 2.4 gave five weeks of advertising spend and sales, in ₹ thousand: spend of 10, 20, 30, 40 and 50, and sales of 25, 35, 38, 52 and 60. The correlation was 0.986. Regression goes a step further and fits a line, written

predicted sales = intercept + slope × spend

The intercept is the predicted value when spend is zero. The slope is how much the prediction changes for each one-unit increase in spend. Many lines could be drawn through the dots. Which is best? The standard answer is least squares: choose the line that makes the sum of the squared vertical gaps between the points and the line as small as possible. The gaps are called residuals, so least squares minimises the sum of squared residuals. Squaring does two jobs. It stops positive and negative gaps from cancelling, and it punishes big misses more than small ones.

There is a neat formula for a single predictor. The slope is the sum of the products of deviations divided by the sum of squared deviations of the predictor, and the intercept makes the line pass through the point of averages.

1. From Lesson 2.4, the sum of products of deviations is 870, and the sum of squared spend deviations is 1,000. 2. The slope is 870 ÷ 1,000 = 0.87. 3. The means are 30 for spend and 42 for sales, so the intercept is 42 − 0.87 × 30 = 15.9.

The fitted line is predicted sales = 15.9 + 0.87 × spend. In words: with no advertising the shop would still sell about ₹15,900, and each extra ₹1,000 of advertising goes with about ₹870 of extra sales. The fitted values for spend of 10, 20, 30, 40 and 50 are 24.6, 33.3, 42.0, 50.7 and 59.4. Against the actual sales of 25, 35, 38, 52 and 60, the residuals are 0.4, 1.7, −4.0, 1.3 and 0.6.

Predict first: this model is used to predict sales when spend is 35. What would you say, and how much would you trust a prediction for spend of 100? Decide before reading on.

For spend of 35 the prediction is 15.9 + 0.87 × 35 = 46.35, about ₹46,000, a sensible figure inside the range of the data. For spend of 100 the line gives 102.9, but you should trust it much less, because the data stops at 50. Using a model outside the range it was fitted on is called extrapolation, and the straight-line pattern might bend, flatten or reverse where there is no data. Interpolation within the range is usually safe, and extrapolation needs a strong reason.

How well does it fit? Residuals and R-squared

The residuals tell you how far off the model is at each point. A single summary of fit is R-squared, written R², which asks: what share of the variation in sales does the model account for? It compares two quantities.

Total variation (SST): the sum of squared gaps between each sales value and the overall average of sales, the spread you would have if you ignored spend. Here it is 778, from Lesson 2.4. Unexplained variation (SSE): the sum of squared residuals, the spread left after using the line. Here it is 0.4² + 1.7² + 4.0² + 1.3² + 0.6² = 0.16 + 2.89 + 16.00 + 1.69 + 0.36 = 21.10.

R² = 1 − SSE ÷ SST = 1 − 21.10 ÷ 778 = 1 − 0.0271 = 0.973. The model accounts for 97.3 per cent of the variation in sales. For a simple regression, R² equals the square of the correlation, 0.986² = 0.973, which ties this lesson back to Lesson 2.4.

How big should R² be? There is no universal answer. In a physics experiment 0.99 might be expected. Predicting human behaviour, such as whether a customer will repay, an R² of 0.2 can be useful. R² also says nothing about whether the model is right: a curve fitted with a line can have a high R² and still be systematically wrong at the ends. And it cannot decrease when you add a predictor, a flaw that matters a great deal and that we meet shortly.

Two other numbers come with every regression summary.

Residual standard error: the typical size of a residual, the square root of SSE ÷ (n − 2). Here it is √(21.10 ÷ 3) = √7.03 = 2.65, so predictions are typically off by about ₹2,650. Standard error of the slope: how much the slope would vary from sample to sample, which is the residual standard error divided by the square root of the sum of squared spend deviations: 2.65 ÷ √1,000 = 0.084. The t statistic is 0.87 ÷ 0.084 = 10.4, strongly significant. A 95 per cent interval for the slope uses the t multiplier for 3 degrees of freedom, 3.18: 0.87 ± 3.18 × 0.084 = 0.87 ± 0.27, so from 0.60 to 1.14. With only five points the slope is known to within a wide band, and honest reporting gives that band, not just 0.87.

Several predictors: multiple regression

Real outcomes depend on more than one thing. Multiple regression adds predictors:

predicted sales = intercept + b1 × spend + b2 × festival week + b3 × price

Each coefficient is read as the change in the prediction for a one-unit change in that predictor holding the others fixed. That phrase is the whole point. If the coefficient on spend is 0.6 when festival week is in the model, it means each extra thousand of spend goes with 600 rupees of extra sales among weeks that are alike in festival status. The adjustment is what lets regression separate overlapping effects, in the way that the partial correlation of Lesson 2.4 did.

Suppose the fitted model is predicted sales = 12 + 0.8 × spend + 10 × festival, where festival is 1 for a festival week and 0 otherwise. A festival-week variable is a dummy variable: a 0 or 1 flag that represents a category. For a normal week with spend of 30, the prediction is 12 + 24 + 0 = 36. For a festival week with the same spend it is 12 + 24 + 10 = 46. The coefficient of 10 means festival weeks sell about ₹10,000 more than ordinary weeks at the same spend. To include a category with k values, you create k − 1 dummies, leaving one value as the reference group, because the last dummy would be perfectly determined by the others. For city with Pune, Mumbai and Nagpur, you might include flags for Mumbai and Nagpur and let Pune be the reference, so the Mumbai coefficient reads as "Mumbai compared with Pune".

Units and transformations affect interpretation. If a predictor is in rupees and another in lakhs, their coefficients are not comparable by size. Standardising the predictors to the same scale, by subtracting the mean and dividing by the standard deviation, makes coefficients comparable, and Lesson 4.4 will need it. If the outcome is skewed, as with income or house prices, a regression on its logarithm often fits better, and then a coefficient of 0.03 means roughly a 3 per cent rise in the outcome per unit of the predictor.

Reading a software summary

A regression summary is a table, and you should be able to read it line by line. Here is an illustrative summary for the model predicted sales = 12 + 0.8 × spend + 10 × festival, fitted to 30 weeks. Term · Coefficient · Standard error · t statistic · What it tells youIntercept · 12.0 · 4.1 · 2.9 · Sales in a normal week with no advertising Spend · 0.80 · 0.12 · 6.7 · Extra sales per extra rupee, festival status held fixed Festival · 10.0 · 3.2 · 3.1 · Extra sales in a festival week at the same spend Divide each coefficient by its standard error and you get the t statistic, exactly as in Lesson 2.3: 0.80 ÷ 0.12 = 6.7 and 10.0 ÷ 3.2 = 3.1. With 27 degrees of freedom, a t statistic beyond about 2.05 is significant at 5 per cent, so both predictors clearly matter. A 95 per cent interval for the spend coefficient is 0.80 ± 2.05 × 0.12, from 0.55 to 1.05. The honest sentence is not "spend adds 0.8" but "each extra thousand of spend goes with between 550 and 1,050 rupees of extra sales, other things equal". Note, too, that the intercept's interval is wide: with a standard error of 4.1 around 12, the baseline is poorly pinned down, and nobody should plan around it.

Finally, look at the units before the numbers. If spend is in rupees thousand and sales are too, a coefficient of 0.8 means each rupee of advertising goes with 80 paise of sales, a return below one rupee that should make a manager ask what else the advertising does, such as building awareness that pays later. A model can be statistically clean and commercially incomplete, and noticing that is part of the analyst's job.

Adjusted R-squared: the honest version

Here is the flaw promised above. Adding a predictor to a regression can never reduce R², even if the new column is pure noise, because the line can always use it a little to fit the sample better. So a model stuffed with irrelevant columns reports a higher R² and looks better when it is worse. Adjusted R-squared fixes this by penalising each extra predictor:

adjusted R² = 1 − (1 − R²) × (n − 1) ÷ (n − p − 1)

where n is the number of rows and p is the number of predictors. It can fall when you add a useless predictor, and it rises only if the new predictor improves the fit by more than chance would.

A worked example. A model with 2 predictors fitted to 20 rows has R² of 0.80. The adjusted value is 1 − 0.20 × 19 ÷ 17 = 1 − 0.2235 = 0.776. Now add five irrelevant predictors, so p is 7. The R² creeps up to 0.81, which looks like an improvement. But the adjusted value is 1 − 0.19 × 19 ÷ 12 = 1 − 0.3008 = 0.699. The honest measure fell from 0.776 to 0.699, and the model got worse.

Adjusted R² is a better guide than R², but it is still computed on the data used to fit the model. The final judge of a model is how it performs on rows it has never seen, which Module 6 develops. A good habit is to prefer the simplest model whose adjusted R² is about as good as the rest.

Fitting a regression in Python

Two libraries cover most needs. statsmodels gives the full statistical summary that you have just learned to read, and scikit-learn fits the model for prediction.

python import numpy as np import statsmodels.api as sm

spend = np.array([10, 20, 30, 40, 50]) sales = np.array([25, 35, 38, 52, 60])

X = sm.addconstant(spend) # adds the intercept column fit = sm.OLS(sales, X).fit()

print(fit.params) # intercept 15.9 and slope 0.87 print(fit.rsquared, fit.rsquaredadj) print(fit.bse) # standard errors of the coefficients print(fit.confint()) # 95% intervals print(fit.predict([[1, 35]])) # prediction at spend 35: 46.35

The summary() of the fit prints a table with the coefficients, standard errors, t statistics, p-values, R² and adjusted R². You now know what each of them means. · Correlation (Lesson 2.4) · Simple regressionGives · One number between −1 and 1 · An equation for prediction Treats the columns · Symmetrically · One as the outcome, one as the predictor Units · None · Slope has units of outcome per unit of predictor Extends to · Pairs of columns · Many predictors at once Link · r squared is the regression R² · The slope equals r times the ratio of the spreads ## The assumptions behind the numbers

The coefficients are valid for any data, but the standard errors, p-values and intervals depend on assumptions about the residuals.

Linearity: the true relationship is roughly a straight line in the predictors. Independence: the residuals are not related to each other, which fails in time series. Constant variance: the residuals have the same spread at every level of the fitted values. Roughly bell-shaped residuals: this matters mostly for small samples and for intervals. No perfect redundancy: predictors are not exact copies of each other.

Lesson 4.3 shows how to check each one with pictures and numbers, and what to do when one fails. For now, remember that a regression always returns an answer, whether or not the assumptions hold, so a table of coefficients is not evidence that the model is sound.

Common mistakes

Treating R² as proof of a good model. It measures fit to this sample, not truth or usefulness. Reading a coefficient as a cause. It is an association, adjusted for the other predictors. Lesson 2.4's cautions still apply. Extrapolating. The line may not continue beyond the data. Adding predictors until R² rises. Use adjusted R² and a holdout. Ignoring units. A coefficient per rupee and one per lakh differ by a factor of 100,000. Forgetting the reference group of a dummy. Each dummy coefficient is relative to the omitted category. Reading the intercept as meaningful when zero is outside the data. A person of age zero or a loan of zero rupees may not exist in your table. Reporting the slope without an interval. With few rows, the band can be wide.

Key takeaways

Linear regression fits the line that minimises the sum of squared residuals. The slope is the change in the prediction per unit of the predictor. For one predictor, slope = sum of products of deviations ÷ sum of squared deviations of the predictor, and the line passes through the point of averages. Here that gives 15.9 + 0.87 × spend. R² = 1 − SSE ÷ SST is the share of variation explained. It equals the squared correlation in simple regression and never falls when predictors are added. Adjusted R² = 1 − (1 − R²)(n − 1)÷(n − p − 1) penalises extra predictors and can fall, so use it to compare models. In multiple regression each coefficient is the effect of one predictor with the others held fixed. Dummy variables carry categories, one less than the number of categories. Check the residual standard error, interval for the slope and assumptions, and do not extrapolate.

Check yourself

1. A fitted line is sales = 15.9 + 0.87 × spend. What does the 0.87 mean? Each extra ₹1,000 of spend goes with about ₹870 more sales, in the range of the data. 2. A model has SST = 500 and SSE = 100. What is R²? 1 − 100 ÷ 500 = 0.80. 3. R² is 0.60 with 25 rows and 3 predictors. What is the adjusted R²? 1 − 0.40 × 24 ÷ 21 = 1 − 0.457 = 0.543. 4. You add a column of random numbers to a model. What happens to R² and to adjusted R²? R² cannot fall and usually rises a little. Adjusted R² usually falls. 5. A model for house prices includes dummies for Mumbai and Nagpur. What does the Nagpur coefficient of −3 lakh mean? Nagpur houses are about ₹3 lakh cheaper than Pune houses, the reference city, when the other predictors are equal.

Try this

Use a small dataset with a numeric outcome and at least three predictors: house prices, fuel use, sales by week, or any open data. Set aside forty-five minutes.

1. Fit a simple regression of the outcome on one predictor by hand for five of the rows, using the sums of products and squares. Check the slope and intercept against software. 2. Compute the residuals, SSE, SST and R² for those five rows. 3. Fit a multiple regression with all your predictors using statsmodels. Write each coefficient in a sentence, with units and with an interval. 4. Add a predictor made of random numbers, refit, and watch what happens to R² and adjusted R². Use the simulator to see the penalty for your number of rows. 5. Choose a prediction inside your data's range and one well outside it. Compare how much you trust each. 6. Write a four-sentence summary: the strongest predictor, how well the model fits and how you know, one result you would not trust and why.

A good result is a short note in which every coefficient carries units and an interval, and the fit is judged by adjusted R² and not by R² alone. The next lesson adapts the same machinery to outcomes that are yes or no, such as default or churn.

Tags: Data Science Linear regression R-squared Adjusted R-squared Modelling