Lesson 6.2: Classification metrics: precision, recall, ROC, Gini, KS, lift
Learn the confusion matrix, precision, recall, ROC and AUC, plus the Gini, KS and lift measures credit and marketing teams use, and when each one misleads.
By the end of this lesson you will be able to build a confusion matrix, compute accuracy, precision, recall, specificity and F1 from it, explain why accuracy is often the wrong score, read a ROC curve and an AUC as a ranking probability, convert AUC to a Gini coefficient, compute the KS statistic from a decile table, and read gain and lift charts to say how much a model concentrates the risk. You will do all of this by hand on a table of 5,000 loans with 301 defaults, so you can see every number come from a count. You should have read Lessons 4.2 and 6.1, since the metrics here score the probabilities a classifier produces, on data it has not seen. The mathematics is ratios, with one tiny counting argument for AUC.
A classifier gives each case a score or a probability. To judge it you must say what a good score is, and the answer depends on the decision it supports. A fraud team that can review 200 alerts a day cares about different things from a lender that must decline or approve every applicant, and from a marketer choosing whom to email. Different metrics answer different questions, and a data scientist who knows only accuracy will pick the wrong model more often than one who knows five.
The confusion matrix
Everything starts with a table that counts how a model's yes-or-no decisions compare with reality. Take 1,000 loan applicants, 60 of whom will default. At a cutoff of 0.2 the model flags 150 applicants as likely defaulters. Of the 150 flagged, 41 do default and 109 do not. Of the 850 not flagged, 19 default and 831 do not.
· Actually defaults · Actually repays · TotalFlagged as default · 41 (true positives) · 109 (false positives) · 150
Not flagged · 19 (false negatives) · 831 (true negatives) · 850
Total · 60 · 940 · 1,000
The four cells have names. A true positive is a defaulter correctly flagged. A false positive is a good borrower wrongly flagged, a false alarm. A false negative is a defaulter missed. A true negative is a good borrower correctly passed. Every classification metric is a ratio of these four counts.
Metric · Formula · Value here · The question it answersAccuracy · (TP + TN) ÷ all · (41 + 831) ÷ 1,000 = 87.2% · How often is the decision right?
Precision · TP ÷ (TP + FP) · 41 ÷ 150 = 27.3% · Of those flagged, how many really default?
Recall (sensitivity, true positive rate) · TP ÷ (TP + FN) · 41 ÷ 60 = 68.3% · Of the defaulters, how many do we catch?
Specificity (true negative rate) · TN ÷ (TN + FP) · 831 ÷ 940 = 88.4% · Of the good borrowers, how many do we pass?
False positive rate · FP ÷ (FP + TN) · 109 ÷ 940 = 11.6% · Of the good borrowers, how many do we wrongly flag?
F1 score · 2 × precision × recall ÷ (precision + recall) · 2 × 0.273 × 0.683 ÷ 0.956 = 0.39 · A single balance of precision and recall
Predict first: a model that flags nobody at all, predicting "repays" for everyone, is applied to the same 1,000 applicants. What is its accuracy, and what does that say about accuracy as a measure? Decide before reading on.
Its accuracy is 940 ÷ 1,000 = 94 per cent, higher than the 87.2 per cent of the useful model, and it catches no defaulters at all. This is the accuracy trap. When the outcome is rare, the majority answer scores high on accuracy while being useless. For rare events, precision and recall, which concentrate on the rare class, are far more informative. Recall of 0 for the "flag nobody" model exposes it at once.
Precision depends on how common the event is
Recall and specificity describe how the model treats each kind of case, and do not depend on how common the cases are. Precision does, which connects straight back to Bayes' rule in Lesson 2.1. A model with 90 per cent recall and 95 per cent specificity has a precision of only 15.4 per cent when 1 per cent of cases are positive, 53.5 per cent at 6 per cent, 81.8 per cent at 20 per cent and 94.7 per cent at 50 per cent. The same model, with the same recall and specificity, looks very different in different populations. A model validated on a balanced sample, with half of the cases positive, will disappoint badly when the real rate is 2 per cent.
From one cutoff to all cutoffs: the ROC curve
The confusion matrix describes one cutoff. A probability model can be used at any cutoff, and a different one gives a different matrix. The ROC curve, short for receiver operating characteristic, shows them all at once. For each cutoff it plots the true positive rate, which is recall, on the vertical axis against the false positive rate on the horizontal axis. A lenient cutoff, flagging many, gives high recall and a high false positive rate, which is the top right of the plot. A strict cutoff gives the bottom left. A useless model, which ranks at random, lies along the diagonal. A perfect one rises straight to the top left corner.
From Lesson 4.2, the same applicants give three cutoffs: at 0.5, recall is 23 per cent and the false positive rate is 0.6 per cent. At 0.2, recall is 68 per cent and the false positive rate is 11.6 per cent. At 0.1, recall is 90 per cent and the false positive rate is 29.4 per cent. Each is a point on the curve, and joining many such points draws it.
AUC: the area, and what it means
The area under the ROC curve, or AUC, summarises the whole curve in one number from 0.5, no better than chance, to 1.0, perfect. It has a clean meaning that makes it easy to explain: the AUC is the probability that a randomly chosen defaulter gets a higher score than a randomly chosen good borrower. An AUC of 0.76 means that in 76 per cent of such pairs the model ranks the defaulter higher.
You can calculate it by counting pairs. Suppose three defaulters score 0.9, 0.7 and 0.4, and four good borrowers score 0.8, 0.5, 0.3 and 0.2. There are 3 × 4 = 12 pairs. The defaulter at 0.9 beats all four good borrowers. The defaulter at 0.7 beats three of them, losing only to 0.8. The defaulter at 0.4 beats two, 0.3 and 0.2. In all, 4 + 3 + 2 = 9 of the 12 pairs are ordered correctly, so AUC = 9 ÷ 12 = 0.75.
Credit risk teams often quote the Gini coefficient, a rescaling of the AUC: Gini = 2 × AUC − 1. It runs from 0 for a random model to 1 for a perfect one. The AUC of 0.75 above gives a Gini of 0.50. The choice between them is a convention: they carry the same information, and a team that says "Gini 0.52" means an AUC of 0.76.
Two properties of AUC are worth knowing. It depends only on the ranking of the scores, so any steadily increasing transformation of the scores leaves it unchanged, which means AUC says nothing about whether the probabilities themselves are accurate, and Lesson 6.3 covers that separately. And with a very rare outcome it can look comfortable while the model is poor at the top of the list, because the false positive rate is measured against a huge number of good borrowers. For such problems, the precision-recall curve is a better companion: it plots precision against recall, and its baseline is the prevalence, not the diagonal.
The KS statistic: how far apart are the two groups?
A measure common in lending is the Kolmogorov-Smirnov statistic, or KS. Sort all borrowers by score from riskiest to safest. At each point, compute the cumulative share of all defaulters and the cumulative share of all good borrowers seen so far. The KS is the largest gap between the two cumulative shares. The bigger the gap, the better the model separates the groups.
Here is the same 5,000-loan table, with 301 defaulters and 4,699 good borrowers, divided into ten equal groups of 500, the riskiest group first.
Decile · Loans · Defaulters · Good · Default rate · Cumulative defaulters · Cumulative good · Gap1 (riskiest) · 500 · 95 · 405 · 19.0% · 31.6% · 8.6% · 22.9
2 · 500 · 60 · 440 · 12.0% · 51.5% · 18.0% · 33.5
3 · 500 · 45 · 455 · 9.0% · 66.4% · 27.7% · 38.8
4 · 500 · 30 · 470 · 6.0% · 76.4% · 37.7% · 38.7
5 · 500 · 25 · 475 · 5.0% · 84.7% · 47.8% · 36.9
6 · 500 · 18 · 482 · 3.6% · 90.7% · 58.0% · 32.7
7 · 500 · 12 · 488 · 2.4% · 94.7% · 68.4% · 26.3
8 · 500 · 8 · 492 · 1.6% · 97.3% · 78.9% · 18.5
9 · 500 · 5 · 495 · 1.0% · 99.0% · 89.4% · 9.6
10 (safest) · 500 · 3 · 497 · 0.6% · 100% · 100% · 0
Work out the third row: after three deciles, 95 + 60 + 45 = 200 defaulters have been seen, which is 200 ÷ 301 = 66.4 per cent of all of them, and 405 + 440 + 455 = 1,300 good borrowers, which is 27.7 per cent of all. The gap is 66.4 − 27.7 = 38.8 percentage points. That is the largest in the table, so the KS is 38.8, often quoted as 38.8 per cent. Credit teams commonly regard a KS in the 30s as acceptable for many unsecured portfolios and above 40 as good, though the right level depends on the product, so treat such bands as rough conventions.
The same table gives the AUC. Joining the points of cumulative good against cumulative defaulters and taking the area with the trapezium rule gives an AUC of 0.758, and so a Gini of 2 × 0.758 − 1 = 0.516. The KS, AUC and Gini are three summaries of the same separation, which is why they tend to move together.
Gain and lift: what a model does for a campaign
Business users find the decile table most useful when it is read as a targeting tool. The cumulative gain answers: if we act on the riskiest 20 per cent of the book, what share of all defaulters do we reach? From the table, the top two deciles hold 51.5 per cent of all defaulters. A random 20 per cent would hold 20 per cent. The top 30 per cent holds 66.4 per cent.
The lift in a decile is its default rate divided by the overall default rate. The overall rate is 301 ÷ 5,000 = 6.02 per cent. The riskiest decile has a rate of 19.0 per cent, so its lift is 19.0 ÷ 6.02 = 3.16: it holds defaulters at 3.2 times the average rate. The second decile has a lift of 1.99 and the third of 1.50. The fourth is about 1.0, at the average. The safest decile has a lift of 0.10, a tenth of the average rate.
Lift is the natural language of marketing and collections. A collections team with the capacity to call 30 per cent of overdue accounts, ordered by the model, reaches 66 per cent of the accounts that will go bad, at lower cost than calling at random. A marketing team that emails only the top two deciles of a response model reaches the customers with the highest chance of responding. In both, the question is not "how accurate is the model?" but "how much better is this than choosing at random, at the depth we can afford?", and gain and lift answer it directly.
Choosing a metric
Situation · Useful metrics · WhyRare event, a cost for each missed case · Recall, precision, PR curve · Focus on the rare class
Ranking applicants or customers · AUC, Gini, KS · Judge the ordering, independent of any cutoff
Campaign or collections with limited capacity · Gain and lift at the capacity you have · Directly measures the benefit of ordering
A decision at a fixed cutoff · The confusion matrix, cost at that cutoff · The actual outcomes of the rule
Balanced classes, equal costs · Accuracy, F1 · Rare situation where accuracy is fair
Using the probabilities themselves, for pricing · Log loss, Brier score, calibration · Next lesson
Several metrics can be combined. A typical credit report states the AUC or Gini and the KS on the test set, with their margins from Lesson 6.1, the lift in the top deciles, and the confusion matrix at the chosen cutoff. A model that wins on one measure and loses on another needs a conversation about which decision matters.
In Python, the pieces are a few calls.
python
import numpy as np
import pandas as pd
from sklearn.metrics import (confusionmatrix, precisionscore, recallscore, f1score,
rocaucscore, roccurve, precisionrecallcurve)
probs = model.predictproba(Xtest)[:, 1]
flag = probs >= 0.2
tn, fp, fn, tp = confusionmatrix(ytest, flag).ravel()
print(precisionscore(ytest, flag), recallscore(ytest, flag), f1score(ytest, flag))
auc = rocaucscore(ytest, probs)
gini = 2 auc - 1
fpr, tpr, cutoffs = roccurve(ytest, probs)
ks = (tpr - fpr).max() # the KS statistic, as a fraction
decile table: gain and lift
df = pd.DataFrame({"p": probs, "y": ytest})
df["decile"] = pd.qcut(df["p"].rank(method="first", ascending=False), 10, labels=range(1, 11))
t = df.groupby("decile", observed=True)["y"].agg(loans="count", defaulters="sum")
t["rate"] = t["defaulters"] / t["loans"]
t["lift"] = t["rate"] / df["y"].mean()
t["cumgain"] = t["defaulters"].cumsum() / t["defaulters"].sum()
Common mistakes
Reporting accuracy for a rare event. A model that flags nobody can score 94 per cent.
Quoting precision without the prevalence. The same model has very different precision in different populations.
Validating on a balanced sample. Metrics such as precision will not carry over to the real rate.
Using AUC alone. It ignores calibration and can hide poor performance at the top of the list.
Reading ROC curves for rare events without a precision-recall curve. The false positive rate flatters.
Comparing AUCs that differ by less than their margin. Use the test-set margin of Lesson 6.1.
Choosing a cutoff by default. The right cutoff depends on costs, as Lesson 6.3 shows.
Computing KS or lift on the training data. It measures memory, not skill.
Key takeaways
The confusion matrix counts true and false positives and negatives. Accuracy, precision, recall, specificity and F1 are ratios of those counts.
Accuracy misleads on rare events: predicting "no" for everyone scores 94 per cent at a 6 per cent default rate. Use precision and recall.
Precision depends on prevalence: a model with 90 per cent recall and 95 per cent specificity has 15 per cent precision at a 1 per cent event rate.
AUC is the probability that a random positive outranks a random negative. Gini = 2 × AUC − 1. Both measure ranking, not calibration.
KS is the largest gap between the cumulative shares of positives and negatives, 38.8 in the worked table. Gain and lift show what ordering buys at the depth you can act on.
Choose the metric from the decision, and report it on held-out data with its uncertainty.
Check yourself
1. A model flags 80 cases, of which 20 are real, out of 100 real cases in 2,000. What are precision and recall?
Precision 20 ÷ 80 = 25 per cent. Recall 20 ÷ 100 = 20 per cent.
2. A model's AUC is 0.82. What is its Gini, and what does 0.82 mean?
Gini = 2 × 0.82 − 1 = 0.64. A random positive scores higher than a random negative 82 per cent of the time.
3. In a decile table, the riskiest decile has a default rate of 15 per cent and the overall rate is 5 per cent. What is its lift?
15 ÷ 5 = 3.0.
4. Why can a model with a high AUC still give poor probability estimates?
AUC depends only on the ranking, so probabilities can be too high or too low across the board and AUC stays the same.
5. Cumulative shares of defaulters and good borrowers in the top 20 per cent are 50 per cent and 15 per cent. What is the gap at that point, and what is the KS if it is the largest?
The gap is 35 points, so the KS is 35.
Try this
Use a classifier you have built, or fit a logistic regression on any dataset with a rare yes-or-no outcome. Set aside forty-five minutes.
1. Score the model on a held-out set. Compute the confusion matrix at a cutoff of 0.5 and at the cutoff that flags the riskiest 20 per cent.
2. Compute accuracy, precision, recall, specificity and F1 at each cutoff, by hand from the counts. Then compute the accuracy of predicting the majority class for everyone.
3. Compute the AUC and the Gini. Check the AUC by counting pairs on a small sample of ten cases.
4. Build a decile table with the number of loans, defaulters, rates, cumulative shares and lift. Find the KS as the largest gap.
5. Plot the cumulative gain against random ordering, and write the sentence "acting on the riskiest 30 per cent reaches x per cent of defaulters".
6. Use the prevalence simulator to see what your model's precision would be if the event were twice as common, and half as common.
7. Write a short note: the best metric for the decision, the numbers, their margins, and what a manager should do at your chosen cutoff.
A good result is a one-page summary with a confusion matrix, AUC, KS, a lift table and one sentence about the decision. The next lesson looks at the metrics for predicting numbers, the question of whether probabilities can be believed, and how to pick a cutoff from costs.
Tags: Data Science Classification metrics ROC AUC Gini coefficient Lift chart