Lesson 4.2: Logistic regression for yes-or-no outcomes
Learn how logistic regression turns inputs into a probability, how to read odds ratios, and how a cutoff trades missed cases against false alarms.
By the end of this lesson you will be able to explain why a straight line is the wrong model for a yes-or-no outcome, read a logistic regression as a probability, convert between probability, odds and log-odds, interpret a coefficient as an odds ratio without the usual mistake, and pick a cutoff by weighing the cost of missed cases against false alarms. You will work with a small default model whose numbers you can verify by hand. You should have read Lesson 4.1, since logistic regression keeps the same linear core, and Lesson 2.1 for probability and Bayes' rule. The mathematics is exponentials and logarithms used lightly, and every step is shown with numbers.
A great many business questions have two possible answers: will this borrower default, will this customer leave, is this transaction fraud, will this email be opened. The outcome is a 0 or a 1. Predicting it is called classification, and the standard first tool is logistic regression. Despite its name, it is a classification method, not a regression of a number. It is fast, it produces genuine probabilities, its coefficients can be explained to a regulator, and many credit scorecards in production today are logistic regressions. Even when you later use fancier models, you will use logistic regression as the baseline to beat.
Why a straight line fails
Suppose you try to predict default, coded as 1 for default and 0 for repaid, with ordinary linear regression on the number of missed payments. The fitted line would give a prediction for each borrower. But a line keeps going. For a borrower with many missed payments it might predict 1.4, and for one with none it might predict −0.2. A probability cannot be above 1 or below 0, so these numbers have no meaning. Also, the relationship is rarely a straight line: going from zero to one missed payment may change the chance of default far more than going from nine to ten, once the risk is already high.
We want a model whose output always lies between 0 and 1, and that bends so that effects flatten at the extremes. The solution is to keep the linear part of regression, and pass its result through a curve that squashes any number into the range 0 to 1. That curve is the logistic function, or sigmoid:
probability = 1 ÷ (1 + e^(−z))
where z is the linear part, intercept plus coefficients times inputs, and e is the constant 2.718. When z is very negative the probability is near 0, when z is 0 it is exactly 0.5, and when z is very positive it is near 1.
Probability, odds and log-odds
Logistic regression is built on a different way of expressing chance. The odds of an event are the probability that it happens divided by the probability that it does not. A probability of 0.2 gives odds of 0.2 ÷ 0.8 = 0.25, often said as "1 to 4". A probability of 0.5 gives odds of 1, and a probability of 0.75 gives odds of 3. The log-odds, or logit, is the natural logarithm of the odds. Log-odds are negative for probabilities below one half, zero at one half and positive above.
Probability · Odds · Log-odds0.10 · 0.111 · −2.20
0.25 · 0.333 · −1.10
0.50 · 1 · 0
0.75 · 3 · 1.10
0.90 · 9 · 2.20
The reason for this change of scale is that log-odds run from minus infinity to plus infinity, so a straight-line formula can live there without ever breaking the rules of probability. The logistic regression model says:
log-odds of the event = intercept + b1 × input1 + b2 × input2 + ...
Everything else follows from converting between these three scales. The linear part z is the log-odds, and the logistic function turns it back into a probability.
A worked example: predicting default
A lender fits a logistic model for default using two inputs: the number of missed payments in the past year and annual income in lakh rupees. The fitted model is:
log-odds of default = −2.2 + 0.9 × missed payments − 0.15 × income
Take a borrower with 2 missed payments and an income of 6 lakh. The log-odds are −2.2 + 0.9 × 2 − 0.15 × 6 = −2.2 + 1.8 − 0.9 = −1.3. The odds are e^(−1.3) = 0.273. The probability is 1 ÷ (1 + e^1.3) = 1 ÷ (1 + 3.669) = 0.214, or 21.4 per cent. As a check, odds of 0.273 give a probability of 0.273 ÷ 1.273 = 0.214.
Holding income at 6 lakh, vary the missed payments. With none, the log-odds are −3.1 and the probability is 4.3 per cent. With one it is −2.2, giving 10.0 per cent. With two, 21.4 per cent. With three, log-odds of −0.4 give 40.1 per cent. With four, log-odds of 0.5 give 62.2 per cent.
Notice that the jumps in probability are 5.7, 11.4, 18.7 and 22.1 points. They grow, then would shrink again as the probability approaches 100 per cent. The coefficient 0.9 is constant on the log-odds scale, but its effect on the probability depends on where you start. That is the main difference from linear regression, where a coefficient has the same effect everywhere.
Reading the coefficients: odds ratios
A coefficient in a logistic regression is a change in log-odds, which is awkward to say aloud. Take the exponential and it becomes an odds ratio: the factor by which the odds are multiplied for a one-unit increase in the input. For missed payments, e^0.9 = 2.46, so each extra missed payment multiplies the odds of default by about 2.46. For income, e^(−0.15) = 0.861, so each extra lakh of income multiplies the odds by 0.861, a fall of about 14 per cent. An odds ratio above 1 raises the odds, below 1 lowers them, and exactly 1 means no effect.
Predict first: a borrower's chance of default is 10 per cent. They miss one more payment, which multiplies their odds by 2.46. Is their new probability 24.6 per cent, more than that, or less? Decide before reading on.
It is 21.5 per cent, less than the naive 24.6. The odds at 10 per cent are 0.111. Multiplied by 2.46 they become 0.273, and a probability for those odds is 0.273 ÷ 1.273 = 21.5 per cent. The common mistake is to read an odds ratio as a risk ratio and multiply the probability. The two are close only when the event is rare. When the base probability is high, an odds ratio of 2.46 moves the probability far less than a factor of 2.46, because probabilities cannot exceed 100 per cent.
How the model is fitted
Linear regression minimises squared residuals. Logistic regression cannot, because the outcomes are 0 and 1, so it uses maximum likelihood: choose the coefficients that make the observed outcomes as probable as possible under the model. For each borrower, the model gives a probability of default. If the borrower did default, the contribution is that probability. If they repaid, it is 1 minus that probability. The likelihood is the product of these over all borrowers, and the fit picks the coefficients that maximise it.
Here is a three-borrower example. Model A gives probabilities of default 0.9, 0.2 and 0.6, and the actual outcomes are default, repaid and default. The likelihood is 0.9 × (1 − 0.2) × 0.6 = 0.9 × 0.8 × 0.6 = 0.432. Model B gives 0.6, 0.4 and 0.5 for the same borrowers, so its likelihood is 0.6 × 0.6 × 0.5 = 0.18. Model A explains the outcomes better. In practice we take the logarithm, to turn products into sums, and report the average negative log-likelihood, called log loss: for model A it is −(ln 0.9 + ln 0.8 + ln 0.6) ÷ 3 = (0.105 + 0.223 + 0.511) ÷ 3 = 0.280, and for model B it is 0.572. Lower log loss is better, and it punishes a confident wrong answer severely: predicting 0.01 for an event that happens costs −ln 0.01 = 4.6, far more than predicting 0.4.
Software finds the maximising coefficients by repeated adjustment, and you never do it by hand. What you need to know is that the goal is probabilities that match reality, not just correct labels.
From probability to decision: the cutoff
The model outputs a probability. To act, you need a decision, such as "decline" or "review", so you choose a cutoff: flag everyone with a probability above it. The default of 0.5 is rarely right. It is a convention, not a rule, and the right cutoff depends on the costs.
Imagine 1,000 applicants, of whom 60 will default, a 6 per cent base rate. Three cutoffs give very different results.
Cutoff · Flagged · Defaulters caught · Share of defaulters caught (recall) · Share of flagged who default (precision)0.5 · 20 · 14 · 23% · 70%
0.2 · 150 · 41 · 68% · 27%
0.1 · 330 · 54 · 90% · 16%
At a cutoff of 0.5 the model is cautious, flags only 20 people and is right 70 per cent of the time, but it misses three quarters of the defaulters. At 0.1 it catches 90 per cent of defaulters but flags 330 people, of whom only 16 per cent default, so it turns away 276 good borrowers. Which is better depends on what a missed default costs against what a rejected good customer costs. If one default loses ₹2 lakh and a rejected good borrower costs ₹10,000 of lost profit, flagging aggressively pays. The base rate matters here exactly as in Lesson 2.1: a rare outcome produces many false alarms at low cutoffs.
Lesson 6.2 and 6.3 develop these ideas fully, with precision, recall, ROC curves and cost-based thresholds. For now, remember the principle: a model gives probabilities, a decision rule gives actions, and the two should be judged separately.
Are the probabilities right? Calibration
A model can rank borrowers well and still give probabilities that are off. Calibration asks whether, among borrowers given a 20 per cent chance, about 20 per cent really default. The check is simple: group the borrowers by predicted probability, and compare the average prediction in each group with the actual default rate.
Predicted band · Borrowers · Average predicted · Actual default rateUnder 5% · 500 · 3.0% · 3.4%
5% to 10% · 300 · 7.5% · 7.1%
10% to 20% · 150 · 14.0% · 18.0%
Over 20% · 50 · 30.0% · 41.0%
In the lower two bands the model is close. In the upper two it is too optimistic: borrowers it rates at 30 per cent actually default 41 per cent of the time. Anyone who prices loans from those probabilities would under-charge exactly the riskiest people. Calibration matters whenever the probability itself is used, for pricing, provisioning or expected loss, and matters less when only the ranking is used. If a model is badly calibrated, it can often be repaired by a second small fitting step on held-out data. Lesson 6.3 returns to this.
A related idea is the scorecard. Credit teams often convert log-odds into points, so that a loan officer can add up a score without a calculator. A common scaling sets 600 points at odds of 50 to 1 of repaying, and adds 20 points every time those odds double. A borrower with odds of 100 to 1 would then score 620, and one with 25 to 1 would score 580. The points are only a relabelling of the log-odds, which is why the logistic model sits behind most scorecards. Lesson 8.3 builds a full one.
Logistic and linear regression side by side
· Linear regression · Logistic regressionOutcome · A number · Yes or no (0 or 1)
Predicts · The value of the outcome · The probability of the outcome
Linear part is · The prediction itself · The log-odds
Coefficient means · Change in the outcome per unit · Change in log-odds per unit, or an odds ratio
Fitted by · Least squares · Maximum likelihood
Fit measure · R-squared, residual error · Log loss, ROC and other classification measures
Typical use · Demand, price, loss amount · Default, churn, fraud, response
## Fitting it in Python
python
import numpy as np
import statsmodels.api as sm
from sklearn.linearmodel import LogisticRegression
statsmodels: full statistical summary, odds ratios and intervals
X = sm.addconstant(df[["missedpayments", "incomelakh"]])
fit = sm.Logit(df["default"], X).fit()
print(fit.summary())
print(np.exp(fit.params)) # odds ratios
print(np.exp(fit.confint())) # intervals for the odds ratios
scikit-learn: fit and predict probabilities
model = LogisticRegression(maxiter=1000)
model.fit(df[["missedpayments", "incomelakh"]], df["default"])
probs = model.predictproba(newborrowers)[:, 1] # probability of default
flags = probs >= 0.2 # a chosen cutoff, not 0.5 by habit
Two warnings. Scikit-learn applies a mild penalty to the coefficients by default, a technique called regularisation that Lesson 4.4 explains, so its coefficients differ slightly from statsmodels. And when predictors are on very different scales, standardise them before fitting, so that the fit is stable and comparisons are fair.
Limits and extensions
Logistic regression draws a straight boundary in the space of the inputs, so it cannot capture curved or interacting patterns unless you add them as new columns, which Lesson 4.3 and the feature engineering lesson show. It assumes the log-odds are linear in each input. It can be thrown by predictors that are nearly copies of each other, a problem called multicollinearity that we study next. With very few examples of the rarer outcome, estimates become unstable, and Lesson 7.2 treats imbalanced data. For more than two outcomes there are extensions, such as multinomial logistic regression, that predict one probability per class.
Common mistakes
Using a straight line for a yes-or-no outcome. Predictions go outside 0 and 1.
Reading an odds ratio as a risk ratio. They are close only for rare events.
Using 0.5 as the cutoff by habit. Choose it from the costs and the base rate.
Judging the model by accuracy alone. With a 6 per cent default rate, predicting "no default" for everyone is 94 per cent accurate and useless.
Forgetting that coefficients depend on units. Income in rupees and in lakh give different coefficients.
Treating predicted probabilities as exact. They are estimates, and they should be checked against actual rates, a step called calibration.
Ignoring the base rate. Alerts from a model for a rare event are mostly false alarms, as Lesson 2.1 showed.
Interpreting coefficients causally. Missing payments predict default, but that does not mean forcing a payment to be made changes the risk.
Key takeaways
Logistic regression predicts the probability of a yes-or-no outcome by passing a linear formula through the logistic curve, probability = 1 ÷ (1 + e^(−z)).
The linear part is the log-odds. Convert between probability, odds and log-odds, and remember that log-odds of 0 mean 50 per cent.
A coefficient's exponential is an odds ratio: the factor by which the odds change per unit. Do not multiply a probability by it.
The model is fitted by maximum likelihood, and log loss measures how well the probabilities match the outcomes.
A cutoff turns a probability into an action. Lowering it raises recall and lowers precision, and the right value depends on the costs.
Logistic regression is a strong, explainable baseline. Compare every later classification model with it.
Check yourself
1. A probability is 0.8. What are the odds and the log-odds?
Odds 0.8 ÷ 0.2 = 4. Log-odds ln 4 = 1.39.
2. A model has log-odds of −2.2 + 0.9 × 3 − 0.15 × 6. What is the probability of default?
Log-odds = −2.2 + 2.7 − 0.9 = −0.4, so the probability is 1 ÷ (1 + e^0.4) = 1 ÷ 2.492 = 0.401, about 40 per cent.
3. An odds ratio is 1.5. A person's chance is 40 per cent. What is the new chance?
Odds 0.667 × 1.5 = 1.0, so the probability is 50 per cent.
4. Why is accuracy a poor measure for a 2 per cent event?
Predicting "no" for everyone is 98 per cent accurate and catches nothing.
5. Lowering the cutoff from 0.5 to 0.2 does what to recall and precision?
Recall rises, since more of the real cases are flagged, and precision usually falls, because more of the flagged are false alarms.
Try this
Use a dataset with a yes-or-no outcome and a few numeric predictors, such as a loan default sample, churn data or a marketing response file. Set aside forty-five minutes.
1. Check the base rate of the outcome and write it down. Compute the accuracy of always predicting the more common answer.
2. Fit a logistic regression with two or three predictors in statsmodels. Write each coefficient as an odds ratio with an interval, in one sentence each.
3. Pick one person and compute their log-odds, odds and probability by hand from the coefficients. Check it against predict.
4. Use the odds ratio simulator to see how an odds ratio from your model changes a probability for a low base rate and for a high one.
5. Compute precision and recall at three cutoffs on your data, as in the table of this lesson. Estimate the cost of each kind of error and choose a cutoff.
6. Write a short note: the base rate, the strongest predictor, the cutoff you chose and why, and one thing the model cannot tell you.
A good result is a note that states every effect as an odds ratio with its interval, shows a worked probability for one person and justifies the cutoff with costs. The next lesson looks at what can go wrong inside regression models of both kinds: predictors that overlap, residuals that fan out, and single points that drag the whole fit.
Tags: Data Science Logistic regression Classification Odds ratio Probability