Lesson 8.3: Capstone: a credit-risk scorecard from WOE to monitoring plan
Build a credit-risk scorecard end to end: weight of evidence and IV, a logistic model, points, Gini and KS, a profit-based cutoff and a monitoring plan.
By the end of this lesson you will be able to carry a credit-risk model from a business brief to a monitored decision tool: define the target, bin variables and compute weight of evidence and information value, fit a logistic model on the encoded variables, scale it into scorecard points, judge it with Gini and KS on development and out-of-time samples, choose a cutoff from the economics of the loan, give an applicant the reasons for a decline, and write the monitoring plan that keeps the model honest. You will turn a score of 549 into odds of 8.6 to 1 and a 10.4 per cent chance of default, find that a cutoff of 550 earns ₹132 lakh against ₹101 lakh for approving everyone, and set a Gini floor of 0.34. You should have read the lessons this project draws on: 3.1 and 5.1 for the loan data and weight of evidence, 4.2 for logistic regression, 6.2 and 6.3 for Gini, KS and cutoffs, 6.4 for windows and the population stability index, and 8.2 for monitoring. The mathematics is logarithms, proportions and the logistic curve, all met before.
This is a case study, and everything that has gone before is a part of it. A scorecard is the traditional form of a credit-risk model: a short table of variables, each cut into bands, with a number of points for every band. A clerk can add the points on a form and read off the score, and a regulator can read the table and see what drives a decision. It remains the standard for retail lending because it is transparent, stable, easy to monitor and easy to explain to a customer, and because a logistic regression on carefully encoded variables is hard to beat on small, noisy data. The figures below are illustrative, built on the 5,000 loans of Lesson 3.1 so that the numbers can be followed by hand.
Step 1: The brief and the target
Every project starts with a decision. A lender wants to decide, at application, whether to approve a personal loan. The model must produce a score that ranks applicants by risk, and a cutoff that turns the score into approve or decline. The model will be used for new applicants, so it must be built on information available at the time of application and nothing later, the leakage rule of Lesson 6.4.
The target. Define a bad as a loan that becomes 90 or more days past due within 12 months of disbursal, the performance window, and a good as one that does not. Loans less than 12 months old are excluded because their outcome is not yet known. Loans in an intermediate state, such as one that reaches 30 days past due and recovers, are called indeterminate, and many teams exclude them so the two groups are clearly different. Document each choice, because a different definition gives a different model.
The data. The development sample is the 5,000 loans disbursed in the earlier part of the history, 301 of them bad, a bad rate of 6.0 per cent. An out-of-time sample of 2,000 later loans, with 124 bads, is held back and used only once, at the end, as in Lesson 6.4. The small number of bads, 301, limits how many variables the model can sensibly use, so the plan is a compact scorecard with a few well-chosen characteristics, not dozens.
Predict first: the sample has 301 bads and 4,699 goods. If you used 40 variables, each cut into 5 bands, would the model be more reliable, or less? Decide before reading on.
Less. Forty variables with five bands each is 200 bands, and the 301 bads are spread among them, giving some bands only a handful of bad loans, so their weights of evidence would be noise. A rule of thumb is to keep at least 30 to 50 bads in a band you want to rely on. With 301 bads that points to a few variables and a handful of bands each.
Step 2: Screen and bin the variables
For each candidate, cut it into bins that are sensible for the business, hold enough bads, and, for ordered variables, show a WOE that moves steadily, as in Lesson 5.1. Then compute the weight of evidence and information value. The three that survive screening are monthly income, employment type and the number of missed payments in the last 12 months.
Characteristic · Bin · Loans · Bads · Goods · WOE · IV contributionIncome · Under ₹20,000 · 800 · 76 · 724 · −0.494 · 0.0486
· ₹20,000 to 40,000 · 2,200 · 143 · 2,057 · −0.082 · 0.0031
· ₹40,000 to 80,000 · 1,400 · 63 · 1,337 · 0.307 · 0.0231
· Over ₹80,000 · 600 · 19 · 581 · 0.672 · 0.0407
Employment · Salaried · 3,000 · 135 · 2,865 · 0.307 · 0.0495
· Self-employed · 1,200 · 102 · 1,098 · −0.372 · 0.0391
· Business owner · 800 · 64 · 736 · −0.306 · 0.0171
Missed payments · None · 3,100 · 140 · 2,960 · 0.303 · 0.0500
· One · 1,100 · 80 · 1,020 · −0.202 · 0.0099
· Two or more · 800 · 81 · 719 · −0.565 · 0.0655
The totals are 0.1155 for income, 0.1057 for employment and 0.1254 for missed payments, all in the medium range of Lesson 5.1. Check one row by hand. The two-or-more band holds 719 of 4,699 goods, 15.30 per cent, and 81 of 301 bads, 26.91 per cent. The ratio is 0.569 and its natural log is −0.565, a risky band, and its contribution is (0.1530 − 0.2691) × (−0.565) = 0.0655.
Two candidates were dropped. A random-number column cut into bins gave an IV of 0.002, the level to expect by chance. A "collections flag" showed an IV near 1.9, far beyond the 0.5 mark of suspicion, and proved on inspection to be recorded only after a loan went wrong, so it is a leak and cannot be used at application. The way an analyst reacts to a variable that looks too good is part of the craft.
Two more checks are routine. First, correlation among the three characteristics should be modest, because strongly correlated variables share credit and give unstable coefficients, the multicollinearity of Lesson 4.3. Here the correlations of the encoded variables are all below 0.3. Second, each variable must be explainable: higher income means lower risk, more missed payments means higher risk, and a self-employed borrower has more variable income. A variable that cannot be defended to a customer or a regulator should not be used, however well it scores. Employment type needs a fairness and policy review before use, since some regulators restrict inputs that act as a proxy for protected groups.
Step 3: Fit the model on weight of evidence
Replace each applicant's value of every characteristic by the WOE of its bin. Then fit a logistic regression for the odds of being good on the three encoded variables. Because the WOE is already the log of a ratio of goods to bads, the coefficients come out close to 1, and in the illustrative fit they are 0.9 for income, 0.8 for employment and 1.1 for missed payments, with an intercept of 2.75. The intercept is close to ln(4,699 ÷ 301) = 2.75, the log of the overall odds of good to bad, which is what it should be when every encoded variable averages near zero.
ln(odds of good) = 2.75 + 0.9 × WOE(income) + 0.8 × WOE(employment) + 1.1 × WOE(missed payments)
All coefficients are positive, which is the sign check for this encoding: a higher WOE means a safer bin and should raise the odds of good. A negative coefficient would point to correlated variables fighting over the same signal. Magnitudes far from 1, such as 0.1 or 3, are a warning.
Step 4: Turn the model into points
A model that outputs log-odds is awkward on a form. Scorecards rescale it with two choices fixed by policy: a base score at base odds, and the points to double the odds, or PDO. Here the base score is 600 at odds of 50 to 1 good to bad, and every 20 points doubles the odds. Then
score = offset + factor × ln(odds of good)
where the factor is PDO ÷ ln 2 = 20 ÷ 0.693 = 28.85 and the offset is 600 − 28.85 × ln 50 = 487.12. Check: at odds of 100 to 1, the score is 487.12 + 28.85 × 4.605 = 620, which is 20 points above 600, since the odds doubled.
Because the model is a sum, the score is also a sum. Each bin of each characteristic gets points equal to factor × (coefficient × WOE + intercept ÷ number of characteristics) + offset ÷ number of characteristics, so that the intercept and offset are shared evenly. The result is the scorecard.
Characteristic · Bin · PointsIncome · Under ₹20,000 · 176.0
· ₹20,000 to 40,000 · 186.7
· ₹40,000 to 80,000 · 196.8
· Over ₹80,000 · 206.3
Employment · Salaried · 195.9
· Self-employed · 180.2
· Business owner · 181.8
Missed payments · None · 198.4
· One · 182.4
· Two or more · 170.9
### Scoring one applicant
An applicant earns ₹30,000 a month, is self-employed and has one missed payment in the last year. Their points are 186.7 for income, 180.2 for employment and 182.4 for missed payments, a total of 549.3, or 549 on the form. The odds of good to bad are 50 × 2^((549.3 − 600) ÷ 20) = 50 × 2^(−2.535) = 8.6 to 1, and the chance of default is 1 ÷ (1 + 8.6) = 10.4 per cent. The same arithmetic works in reverse, so a team can speak either in scores or in probabilities.
Reasons for a decline. If the applicant is declined, they are owed the main reasons. Compare each characteristic's points with the best available bin: income lost 206.3 − 186.7 = 19.6 points, missed payments lost 198.4 − 182.4 = 16.0 and employment lost 195.9 − 180.2 = 15.7. The reasons, in order, are "income lower than for most approved applicants", "a missed payment in the last year" and "self-employed income pattern". This is the scorecard version of the SHAP reason codes of Lesson 7.1, and it is exact, because the score is a sum.
Step 5: Judge the model
Cut the development scores into five bands, from riskiest to safest, and count goods and bads in each.
Band · Score · Loans · Bads · Bad rate · Cumulative bads · Cumulative goods · GapA · Under 535 · 400 · 80 · 20.0% · 26.6% · 6.8% · 19.8
B · 535 to 549 · 900 · 85 · 9.4% · 54.8% · 24.2% · 30.7
C · 550 to 564 · 1,500 · 80 · 5.3% · 81.4% · 54.4% · 27.0
D · 565 to 579 · 1,300 · 40 · 3.1% · 94.7% · 81.2% · 13.5
E · 580 and above · 900 · 16 · 1.8% · 100% · 100% · 0.0
The bad rate falls steadily from 20.0 per cent in the riskiest band to 1.8 per cent in the safest, a tenfold range, which is what a working ranking looks like. The KS statistic is the largest gap between the cumulative shares of bads and goods, 30.7 points in band B. Joining the points and taking the area under the ROC curve with the trapezium rule gives an AUC of 0.705 and a Gini of 2 × 0.705 − 1 = 0.41. These are respectable for a three-variable scorecard on small data: a Gini of 0.41 is not exceptional, and the aim of a scorecard is a stable, explainable ranking and not a record score.
The out-of-time test. The held-back 2,000 loans give an AUC of 0.69, a Gini of 0.38 and a KS of 28. The fall of 0.03 from 0.41 is inside the margin of about 0.07 for a sample of this size that Lesson 8.2 explained, so the model has held up. The honest report states the development and out-of-time figures side by side, with their margins. It also checks the fit of the levels: the expected bad rate of each band against the observed rate in the out-of-time sample, using control bands. A model that ranks well but predicts levels badly can be recalibrated; one that ranks badly cannot.
Step 6: Choose the cutoff from the economics
A cutoff converts the score into a decision. Lesson 6.3 showed that the best cutoff comes from costs, not from a default of 50 per cent. Suppose a good loan earns ₹6,000 in profit and a bad loan loses ₹60,000. A band breaks even when the profit on its goods equals the loss on its bads, which happens at a bad rate of 6,000 ÷ (6,000 + 60,000) = 9.09 per cent. Bands with a lower bad rate make money. Band A at 20.0 per cent clearly loses, band B at 9.4 per cent only just loses, and bands C, D and E make a profit.
Work out the total profit for four cutoffs on the development loans.
Approve · Loans approved · Bads · Bad rate · Profit (₹ lakh)Everyone · 5,000 · 301 · 6.0% · 101.3
Bands B to E (score 535 and above) · 4,600 · 221 · 4.8% · 130.1
Bands C to E (score 550 and above) · 3,700 · 136 · 3.7% · 132.2
Bands D and E (score 565 and above) · 2,200 · 56 · 2.5% · 95.0
For example, for bands C to E, the 3,700 approved loans include 136 bads and 3,564 goods, so the profit is 3,564 × ₹6,000 − 136 × ₹60,000 = ₹213.8 lakh − ₹81.6 lakh = ₹132.2 lakh. A cutoff of 550 gives the best profit, ₹132 lakh against ₹101 lakh for approving everyone, an extra ₹31 lakh, while declining 26 per cent of applicants. Raising the cutoff to 565 declines too many good loans and profit falls to ₹95 lakh. The break-even odds are 60,000 ÷ 6,000 = 10 to 1, which is a score of 600 + 20 × log2(10 ÷ 50) = 553.6, so the band-edge cutoff of 550 is close to the theoretical one.
The profit view is a starting point. Real cutoffs also reflect capacity, regulation, pricing by risk, a portfolio target for growth and policy rules that sit around the score, such as minimum income and fraud checks. A common design is three zones: approve above one score, decline below another, and send the middle for manual review.
Step 7: The monitoring plan
A scorecard is not finished at launch. Using Lesson 8.2, the plan is a table with a limit, an action and an owner for every line.
Measure · How often · Expected value · Amber · Red · Action on redScore PSI against development (bands 8, 18, 30, 26, 18%) · Weekly · Below 0.1 · 0.10 to 0.25 · Above 0.25 · Find the cause, review with risk head
Characteristic stability of each input · Monthly · Below 0.1 · 0.10 to 0.25 · Above 0.25 · Check the data feed, then the channel mix
Missing or invalid input share · Daily · Below 1% · 1% to 3% · Above 3% · Stop scoring on the field, raise a data incident
Approval rate and override rate · Weekly · Within 3 points of plan · 3 to 5 points · Above 5 points · Review policy rules and cutoff
90-day delinquency by band at month 6 (early indicator) · Monthly · Inside the control band · One band outside · Two or more outside · Investigate the cohort, consider a tighter cutoff
Bad rate by band at month 12 · Quarterly · Inside the control band · One band outside · Two or more outside · Recalibrate or refit
Gini on matured cohorts · Quarterly · About 0.41 · Below 0.37 · Below 0.34 · Refit or rebuild
The Gini floor of 0.34 comes from the development value of 0.41 less the margin of about 0.07 for a sample of this size, so that a red reading means more than noise. The band shares used for the PSI are the development shares of 8, 18, 30, 26 and 18 per cent. Each control band follows the formula of Lesson 8.2: for band C, with an expected bad rate of 5.33 per cent and a cohort of 1,200 loans, it runs from 4.1 to 6.6 per cent.
The plan also names the owner of each measure, the date of the first review, the rule for a scheduled refit, such as every twelve months, and the process for approving a replacement: a challenger scored in shadow, validated out of time, documented and signed off before it replaces the champion, as in Lesson 8.1.
Step 8: Document and state the limits
The last deliverable is a model document that someone who was not in the room can follow. A good one contains:
1. Purpose and use: the decision, the population, the approved and prohibited uses.
2. Data: sources, the sample, exclusions, the target definition and the windows.
3. Method: variables considered and dropped with reasons, bins, WOE and IV, the model and the scaling.
4. Performance: AUC, Gini, KS and the band table on development and out-of-time samples, with margins.
5. Cutoff and economics: the profit assumptions and the effect of the chosen cutoff.
6. Fairness and policy review: the checks made on inputs and outcomes by group, and the decisions taken.
7. Monitoring plan: the table above, with owners.
8. Limitations: what the model cannot do.
The limitations here are worth stating plainly. The model rests on 301 bads and three characteristics. It was built on approved loans only, so it knows little about the applicants who were declined. Its probabilities hold for the population of the development period, and it will need recalibration if the economy or the channel mix moves. It uses no bureau score or repayment history, which a production scorecard would, and a better model should be expected once they are available. A document that says these things is more trustworthy than one that does not.
python
import numpy as np
import pandas as pd
from sklearn.linearmodel import LogisticRegression
from sklearn.metrics import rocaucscore, roccurve
from sklearn.modelselection import traintestsplit
d = df[["income", "employment", "missedpayments", "default"]].copy()
d["incomebin"] = pd.cut(d["income"], [0, 20000, 40000, 80000, np.inf],
labels=["<20k", "20-40k", "40-80k", ">80k"])
d["missedbin"] = pd.cut(d["missedpayments"], [-1, 0, 1, 99], labels=["0", "1", "2+"])
chars = ["incomebin", "employment", "missedbin"]
train, test = traintestsplit(d, testsize=0.3, stratify=d["default"], randomstate=0)
def woetable(frame, col, target="default"):
t = frame.groupby(col, observed=True)[target].agg(bads="sum", n="count")
t["goods"] = t["n"] - t["bads"]
pg = (t["goods"] + 0.5) / (t["goods"] + 0.5).sum() # +0.5 avoids log(0) in thin bins
pb = (t["bads"] + 0.5) / (t["bads"] + 0.5).sum()
t["woe"] = np.log(pg / pb)
t["iv"] = (pg - pb) t["woe"]
return t
tables = {c: woetable(train, c) for c in chars} # learned on training rows only
print({c: round(t["iv"].sum(), 3) for c, t in tables.items()})
def encode(frame):
return pd.DataFrame({c: frame[c].map(tables[c]["woe"]).astype(float) for c in chars})
good = 1 - train["default"] # model the odds of GOOD, so signs are positive
lr = LogisticRegression(C=1e6, maxiter=1000).fit(encode(train), good)
print(dict(zip(chars, lr.coef[0].round(2))), round(lr.intercept[0], 2))
scale to points: base 600 at 50 to 1, 20 points to double the odds
pdo, base, baseodds = 20, 600, 50
factor = pdo / np.log(2)
offset = base - factor np.log(baseodds)
k = len(chars)
points = {c: factor (lr.coef[0][i] tables[c]["woe"] + lr.intercept[0] / k) + offset / k
for i, c in enumerate(chars)}
def score(frame):
return sum(frame[c].map(points[c]).astype(float) for c in chars)
stest = score(test)
auc = rocaucscore(test["default"], -stest)
fpr, tpr, = roccurve(test["default"], -stest)
print("Gini", round(2 auc - 1, 3), "KS", round((tpr - fpr).max(), 3))
bands and the profit of each cutoff (profit per good Rs 6,000, loss per bad Rs 60,000)
bands = pd.cut(stest, 5)
tab = test.groupby(bands, observed=True)["default"].agg(loans="count", bads="sum")
tab["profitlakh"] = ((tab["loans"] - tab["bads"]) 6000 - tab["bads"] 60000) / 1e5
print(tab)
Common mistakes
Defining the target loosely. The bad definition and windows decide everything. Write them down and apply them consistently.
Too many variables for the bads. With a few hundred bads, keep the scorecard compact and the bands well populated.
Believing a very high IV. A variable that separates bads from goods almost perfectly is usually a leak. Investigate.
Fitting the WOE on all the data. Learn bins and WOE from training rows only, then apply them unchanged.
Unexplainable or unfair variables. A variable you cannot defend, or that acts as a proxy for a protected group, should not be used.
Judging on development data alone. Report an out-of-time result with margins.
Choosing the cutoff by habit. Derive it from profit and loss, then test the sensitivity of the answer to the assumptions.
No monitoring plan. A scorecard without limits, owners and actions will drift unnoticed.
Hiding the limits. State the sample, the missing information and the conditions under which the model holds.
Key takeaways
A scorecard is a table of points per bin, built from weight of evidence and a logistic regression. It is transparent, stable and easy to monitor.
Start with the decision, the target and the windows. With 301 bads, use a few variables and bins that hold enough bads.
WOE = ln(share of goods ÷ share of bads), and IV adds up the evidence. Dropping a suspiciously strong variable, like the collections flag with an IV near 1.9, is part of the work.
Scale with a base score, base odds and PDO. With 600 at 50 to 1 and 20 points to double the odds, a score of 549 means odds of 8.6 to 1 and a 10.4 per cent chance of default.
Judge with the band table, Gini and KS on development and out-of-time samples: Gini 0.41 and 0.38 here, within the margin.
Choose the cutoff from economics. Break-even bad rate = profit per good ÷ (profit per good + loss per bad) = 9.09 per cent, and a cutoff of 550 earns ₹132 lakh against ₹101 lakh for approving everyone.
Write the monitoring plan before launch, with limits that respect noise, an action and an owner, and document the limits of the model.
Check yourself
1. A bin holds 100 of 1,000 goods and 30 of 100 bads. What is its WOE?
ln(0.10 ÷ 0.30) = −1.099. The bin holds relatively more bads, so it is a risky bin.
2. A policy sets 25 points to double the odds. What is the factor?
25 ÷ ln 2 = 36.07 points per unit of log-odds.
3. With a base of 600 at 50 to 1 and 20 points to double the odds, what are the odds at a score of 620, and the probability of default?
20 points above the base doubles the odds to 100 to 1. The probability of default is 1 ÷ 101 = 0.99 per cent.
4. A good loan earns ₹5,000 and a bad loan loses ₹45,000. What is the break-even bad rate?
5,000 ÷ (5,000 + 45,000) = 10 per cent.
5. Gini was 0.41 on development and 0.38 on an out-of-time sample, with a margin of about 0.07. Has the model failed?
No. The fall of 0.03 is inside the margin. Report both values and keep watching the trend.
6. Why does the monitoring plan use a Gini floor of 0.34?
It is the development Gini of 0.41 less the margin of about 0.07, so a reading below it means more than noise.
Try this
Build a compact scorecard on a loan or customer table with a yes-or-no outcome. Set aside two hours.
1. Write the target definition and the windows. List what you exclude and why.
2. Bin at least five candidate variables, compute WOE and IV, and plot the WOE of each ordered variable. Drop any with an IV below 0.02 and examine any above 0.5.
3. Keep three to five variables with sensible, monotonic bins, and check that their encoded values are not strongly correlated.
4. Fit a logistic regression on the WOE-encoded training data. Check that every coefficient is positive and near 1 when you model the odds of good.
5. Scale to points with a base score, base odds and a PDO of your choice. Score a test applicant by hand and check the odds and probability with the simulator.
6. Compute the band table, Gini and KS on a time-based test sample, and write down their margins.
7. Choose a cutoff from a profit per good and a loss per bad. Test how the best cutoff moves if the loss per bad rises by 25 per cent.
8. Write a monitoring table with measures, limits, owners and actions, and a model document with its limitations.
A good result is a scorecard that a colleague can score by hand, a band table with a Gini and its margin, a cutoff justified in rupees and a monitoring plan with a limit and an owner for every line. The final lesson is about the last step of every analysis, which is getting people to understand it and act.
Tags: Data Science Credit scorecard Weight of evidence Gini coefficient Model monitoring