Lesson 7.1: Explaining models: importance, partial dependence and SHAP
Learn to open up a black-box model with permutation importance, partial dependence and SHAP, and turn a prediction into reasons a customer can follow.
By the end of this lesson you will be able to say which features a model relies on, show how a prediction changes as one feature moves, and explain a single decision in terms of the contribution of each input. You will compute a permutation importance from score drops, build a partial dependence curve by hand from four borrowers, and calculate a Shapley value for a two-feature model from first principles, then check that the pieces add up to the prediction. You should have read Lessons 4.2, 5.2 and 6.2, since the explanation methods here are applied to the logistic and boosted models you built, and judged with the metrics you learned. The mathematics is averages, differences and the logistic curve, and every number can be checked by hand.
The previous modules were about building models that predict well. This one is about the other half of the job: understanding and communicating what a model does. A boosted tree with several hundred trees and dozens of features is not something you can read. Yet the people who use it ask reasonable questions. A credit officer wants to know why this applicant was declined. A regulator wants evidence that the model does not lean on a prohibited or unreasonable input. A product manager wants to know which features are worth improving. A data scientist, debugging, wants to find out whether the model has learned something real or a leak from Lesson 6.4. Explanation methods answer these questions, and they are as much a part of delivering a model as accuracy.
What kind of explanation do you need?
Explanations differ in scope, and picking the right one depends on the question.
Question · Scope · Typical methodWhich features does the model rely on overall? · Global · Permutation importance, mean absolute SHAP
How does the prediction change as this feature changes? · Global, one feature · Partial dependence, ICE curves
Why did the model give this particular answer? · Local, one case · SHAP values, reason codes, LIME
Can the whole model be summarised by simple rules? · Global · A surrogate tree or a simple model
Some models explain themselves. A linear or logistic regression is intrinsically interpretable: each coefficient says what a feature does, as in Lessons 4.1 and 4.2. A shallow decision tree can be read as a flowchart. For complex models, such as forests, boosted trees and neural networks, we use post-hoc methods that treat the model as a box and study its behaviour from outside. A recurring theme is a trade-off: complex models are often more accurate and harder to explain, and sometimes the right decision is a simpler model that can be defended, as Lesson 5.3 noted.
Permutation importance: what happens if I scramble this feature?
The idea is simple and reliable. Take a trained model and a held-out set, and record its score, say an AUC of 0.80. Now shuffle the values of one feature across the rows, breaking its link with the outcome while leaving everything else alone, and score again. If the model leaned heavily on that feature, its performance falls a lot. If it ignored the feature, nothing changes. The permutation importance of a feature is the drop in score caused by shuffling it, usually averaged over several shuffles.
Suppose the default model has a held-out AUC of 0.80. Shuffling missed payments drops it to 0.66, a loss of 0.14. Shuffling income drops it to 0.74, a loss of 0.06. Shuffling loan-to-income gives 0.77, a loss of 0.03, age gives 0.795, a loss of 0.005, and tenure leaves it at 0.80.
Permutation importance works with any model and any metric, and because it is computed on held-out data it reflects what the model uses to generalise, not what it memorised. It has a known weakness with correlated features. If income and loan amount carry similar information, shuffling one leaves the other to cover for it, so both look less important than they are. The cure is to group correlated features and shuffle the group, or to treat the importances of correlated features as a combined total.
Predict first: a model uses two nearly identical features, say income in rupees and income in dollars. You shuffle one of them. Will the score drop a lot, a little or not at all? Decide before reading on.
Hardly at all. The model can read the same information from the copy you left alone, so shuffling one feature barely hurts, and the feature looks unimportant even though the model relies on income heavily. This is why a single importance number must be read alongside the correlation structure of the features, which Lesson 4.3's variance inflation factors help you see.
Partial dependence: how does the prediction move with a feature?
Importance tells you that a feature matters. It does not tell you how. For that, vary the feature and watch the prediction. A partial dependence plot, or PDP, fixes the feature of interest at a series of values, sets it that way for every row in the data, and averages the model's predictions. It answers: on average, what would the model predict if everyone had an income of ₹2 lakh? Of ₹4 lakh? And so on.
Take the default model from Lesson 4.2, with log-odds equal to −2.2 + 0.9 × missed payments − 0.15 × income. Use four borrowers with 0, 1, 2 and 3 missed payments. At an income of ₹2 lakh their predicted default probabilities are 7.6, 16.8, 33.2 and 55.0 per cent, and the average is 28.1 per cent. At ₹6 lakh they are 4.3, 10.0, 21.4 and 40.1 per cent, averaging 19.0. At ₹12 lakh they are 1.8, 4.3, 10.0 and 21.4, averaging 9.4. The averages are the partial dependence curve, and they fall as income rises.
The individual curves are called ICE curves, for individual conditional expectation. Each line shows how one borrower's prediction would change as income changes, with their other features held fixed.
ICE curves show what an average can hide. Here every borrower's curve falls, and the one with three missed payments falls by 33.6 points, while the one with none falls by only 5.8. If the individual curves had crossed or moved in opposite directions, the average could have been flat and meaningless. This is the interaction problem of Lesson 4.3 seen from outside: the effect of income depends on the borrower's missed payments. A plot that shows only the average would suggest an effect of the same size for everyone.
PDPs carry a warning. They average over the feature values in the data, and when the feature is correlated with others, they evaluate the model at combinations that never occur: a borrower with an income of ₹2 lakh and a loan of ₹50 lakh, say. Variants called accumulated local effects avoid this by averaging local changes only among realistic neighbours.
SHAP: dividing the credit for one prediction
Importance and partial dependence describe the model as a whole. To explain a single decision we need a way to share out the prediction among the features. The most widely used method is SHAP, short for SHapley Additive exPlanations, which borrows an idea from cooperative game theory: the Shapley value. It answers the question of how to divide a payoff fairly among players whose contributions overlap.
The setting. The model's average prediction over all borrowers is the baseline, say 6 per cent. For one borrower the model predicts 16 per cent. The gap of 10 points must be explained by the features. The Shapley value of a feature is its average marginal contribution over every order in which the features could be revealed. Here is the calculation for two features, missed payments and income. Define v(S) as the model's expected prediction when only the features in S are known and the rest are averaged out.
v(none) = 6, the baseline.
v(missed payments only) = 14.
v(income only) = 9.
v(both) = 16, the actual prediction.
If missed payments is revealed first, it adds 14 − 6 = 8. If income comes first and then missed payments, missed payments adds 16 − 9 = 7. The Shapley value for missed payments is the average of the two orders: (8 + 7) ÷ 2 = 7.5. For income, revealed first it adds 9 − 6 = 3, and revealed second it adds 16 − 14 = 2, so its value is (3 + 2) ÷ 2 = 2.5.
Check the additivity property: the baseline plus the Shapley values equals the prediction, 6 + 7.5 + 2.5 = 16. The values always add up exactly to the gap between the prediction and the baseline, which is the property that makes SHAP so useful for explaining a case.
With many features, the number of orders explodes, so software uses shortcuts. For tree models, a fast exact algorithm called TreeSHAP exists. For others there are sampling-based approximations.
SHAP for a logistic regression, by hand
For a logistic regression the calculation is simple, and it is the best way to see what SHAP is doing. On the log-odds scale, each feature's contribution is its coefficient times the gap between the borrower's value and the average value of the feature. Take the model −2.2 + 0.9 × missed payments − 0.15 × income, with an average of 0.8 missed payments and an average income of ₹8 lakh. The baseline log-odds is −2.2 + 0.9 × 0.8 − 0.15 × 8 = −2.68, which is a default probability of about 6.4 per cent.
A borrower has 3 missed payments and an income of ₹5 lakh. The contribution of missed payments is 0.9 × (3 − 0.8) = +1.98. The contribution of income is −0.15 × (5 − 8) = +0.45, positive because this borrower earns less than average. The prediction in log-odds is −2.68 + 1.98 + 0.45 = −0.25, which is a probability of 1 ÷ (1 + e^0.25) = 43.8 per cent. Computing the model directly, −2.2 + 0.9 × 3 − 0.15 × 5 = −0.25, the same.
Using SHAP
SHAP values are useful at three levels. For one case, sort the contributions and read the top ones as the reasons for the prediction: "this application was rated high risk mainly because of three missed payments in the last year and a lower than usual income." Lenders in many jurisdictions are expected to give applicants the main reasons for an adverse decision, and SHAP, or the coefficients of a logistic model, provide them. For a whole dataset, average the absolute SHAP values of each feature to get a global importance ranking that, unlike impurity importance, has a consistent meaning. And plotting each row's SHAP value for a feature against the feature's value reveals the shape of the relationship, including interactions, as a richer alternative to a PDP.
Method · Scope · Works with · Cost · Main caveatCoefficients · Global · Linear and logistic models · None · Only for intrinsically interpretable models
Impurity importance · Global · Tree models · None · Biased towards features with many values
Permutation importance · Global · Any model · Moderate · Understates correlated features
Partial dependence, ICE · Global, one feature · Any model · Moderate · Averages hide interactions, unrealistic combinations
SHAP · Local and global · Any model, fast for trees · Low for trees, high otherwise · Depends on how the baseline is defined, and on correlated features
LIME · Local · Any model · Low · Unstable: different runs give different answers
Surrogate model · Global · Any model · Low · Only as good as its fit to the model
### From contributions to reason codes
A lender that declines an application usually has to say why, in a few plain lines. SHAP values make this mechanical. For one declined applicant, suppose the contributions in log-odds are: missed payments +1.98, income below average +0.45, loan-to-income ratio +0.30 and tenure with the bank −0.20, with several smaller ones. A positive contribution pushes towards default and a negative one pushes away. Sort the positive contributions from largest to smallest and take the top three or four. The reasons are then "number of missed payments in the last twelve months", "monthly income below the level typical for this product" and "loan amount high relative to income". The tenure factor, which counted in the applicant's favour, is not a reason for the decline and is left out.
Two practical cautions make this safe. First, each feature name must map to a sentence a customer can act on, so a raw feature such as dpd30cnt12m needs a reviewed plain-language label. Second, the reasons must reflect the model that made the decision, on the data it saw. If a column was later corrected, rerun the explanation instead of reusing an old one. Teams often store the SHAP values of every decision at the time it is made, so that a reason can be reproduced months later if a customer or a regulator asks.
Pitfalls
Explanation is not causation. A model that relies on missed payments tells you what predicts default, not what would change it. Forcing a payment to be made would not necessarily change risk.
Explaining a leaky model. If the top feature is a leak from Lesson 6.4, the explanation will faithfully report it. Use explanations as a leak detector: when a surprising feature dominates, investigate.
Correlated features. Credit can be split arbitrarily between near-copies, so group them.
Unstable explanations. Small changes to the data or model can shift importances, so check them across cross-validation folds.
Faithfulness. A surrogate or a simplified story may sound clear without reflecting what the model does.
Audience. A regulator, a customer and a developer need different levels of detail. Write each version for its reader.
python
import numpy as np
import pandas as pd
import shap
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.inspection import permutationimportance, PartialDependenceDisplay
model = GradientBoostingClassifier(learningrate=0.05, nestimators=300, maxdepth=3,
randomstate=0).fit(Xtrain, ytrain)
global: which features does the model use? (on held-out data)
perm = permutationimportance(model, Xvalid, yvalid, scoring="rocauc",
nrepeats=20, randomstate=0)
pd.Series(perm.importancesmean, index=Xvalid.columns).sortvalues(ascending=False)
how does the prediction move with a feature? average curve and individual (ICE) curves
PartialDependenceDisplay.fromestimator(model, Xvalid, ["incomelakh"], kind="both")
local and global with SHAP: contributions in log-odds that add up to the prediction
explainer = shap.Explainer(model, Xtrain)
sv = explainer(Xvalid.iloc[:200])
reasons = pd.Series(sv.values[0], index=Xvalid.columns).sortvalues(key=abs, ascending=False)
print(reasons.head(3)) # the top reasons for one decision
print(np.abs(sv.values).mean(axis=0)) # global importance: mean absolute SHAP
Common mistakes
Reading importance as cause. It measures reliance, not effect.
Using impurity importance for decisions. It favours features with many distinct values. Prefer permutation or SHAP.
Computing importance on training data. It rewards memorisation. Use held-out rows.
Ignoring correlated features. Group them, or treat their importances together.
Trusting one partial dependence line. Check ICE curves for interactions.
Giving customers raw SHAP numbers. Translate the top contributions into plain reasons.
Explaining without checking. If an explanation is surprising, test whether the model or the data is wrong.
Assuming a model that can be explained is fair. Explainability helps you audit a model and does not by itself make it fair.
Key takeaways
Models differ in how they can be explained. Linear models and shallow trees explain themselves, and complex models need post-hoc tools. Choose the scope: global, one feature or one case.
Permutation importance is the drop in held-out score when a feature is shuffled. It works for any model, and understates correlated features.
A partial dependence plot averages predictions as one feature varies, and ICE curves show individual paths. Averages can hide interactions.
A Shapley value is a feature's average marginal contribution over every order of revealing the features. SHAP values add up exactly to the prediction minus the average, 6 + 7.5 + 2.5 = 16 in the example.
For a logistic regression, SHAP in log-odds is the coefficient times the gap between the borrower's value and the average.
Explanations describe the model, not the world. They are a way to find leaks and to give reasons, and need to be tested like any other result.
Check yourself
1. A model's held-out AUC is 0.82, and shuffling feature X drops it to 0.80. What does that suggest?
The model uses X only slightly, or another feature carries the same information. Check for correlated features.
2. Why can a flat partial dependence curve be misleading?
The average may hide individual curves that rise and fall and cancel out. Check ICE curves for interactions.
3. Baseline prediction 5 per cent, with Shapley values +6 and −1 for two features. What is the prediction?
5 + 6 − 1 = 10 per cent.
4. In a logistic model a coefficient is 0.4, the average value of the feature is 2 and a borrower's value is 6. What is the contribution in log-odds?
0.4 × (6 − 2) = +1.6.
5. Why should importance be computed on held-out data?
On training data the model has memorised the rows, so importances reflect memorisation. Held-out data shows what it uses to generalise.
Try this
Use the default model you built in Module 5 or 6, or fit a boosted model on any dataset with a yes-or-no outcome. Set aside forty-five minutes.
1. Compute permutation importance on held-out data with at least ten repeats. Write the top three features and the score each costs.
2. Check whether any two top features are correlated. Shuffle them as a group and compare the result with their separate importances.
3. Draw a partial dependence curve with ICE curves for your most important numeric feature. Say whether the curves move together or cross.
4. Pick one declined or high-risk case. Compute SHAP values, list the top three contributions, and write them as two sentences a customer could understand.
5. Check that the baseline plus the contributions equals the model's output for that case, on the scale you used.
6. For a logistic version of the model, compute the SHAP contribution of one feature by hand as the coefficient times the gap from the average, and check it against the library.
7. Write a short note: what the model relies on, anything surprising that might signal a leak, and one thing you would not claim from these explanations.
A good result is a note with a ranked list of drivers, one plain-language explanation of a single decision, and a clear statement of what the explanations cannot show. The next lesson turns to a practical problem that makes many of these models harder to train: outcomes so rare that the model barely sees them.
Tags: Data Science Model explainability SHAP Partial dependence Permutation importance