Lesson 7.2: Imbalanced data, anomalies and fraud
Learn how to model rare events such as fraud: resampling and class weights, why undersampling distorts probabilities, and how isolation forests find anomalies.
By the end of this lesson you will be able to explain why rare outcomes break the usual habits of modelling, choose among resampling, class weights and threshold changes, correct a probability that undersampling has distorted, flag unusual values with a robust score and an isolation forest, and judge a fraud model by what a review team can actually use. You will work through a table of 100,000 transactions with only 200 frauds, see that a model's "50 per cent" after undersampling means about 2 per cent in the real world, compute an isolation forest's anomaly score from path lengths, and read precision at the depth a team can review. You should have read Lessons 6.2 and 6.3, since rare-event work depends on precision, recall and cost-based cutoffs, and Lesson 2.1 for the base rate fallacy. The mathematics is ratios, odds and one logarithm.
The most valuable events in business are often the rarest. Fraud is a tiny share of transactions. Defaults in a prime portfolio are a few in a hundred. Equipment failures, money-laundering cases and cyber intrusions are rarer still. These are imbalanced problems, in which one class, the majority, vastly outnumbers the other, the minority. Almost every default setting in a machine learning library assumes a fairly balanced problem, so rare-event work needs a different set of habits.
Why imbalance causes trouble
Take a payments company with 100,000 transactions in a day, of which 200 are fraudulent, a rate of 0.2 per cent. Consider what goes wrong.
Accuracy is useless. A model that calls everything genuine is 99.8 per cent accurate and catches nothing, the accuracy trap of Lesson 6.2.
The model has little to learn from. The training loss is dominated by the 99,800 genuine cases. A model that predicts a tiny probability for every transaction already achieves a low loss, and has little incentive to find the 200.
Validation is noisy. A test set of 20,000 rows holds about 40 frauds, so recall rests on 40 cases and has a margin of error of around 15 points, as Lesson 6.1 warned.
Alerts are mostly false. Even a very good model produces mostly false alarms among its flags, because of the base rate, as Lesson 2.1 showed.
The first rule is therefore to change the questions you ask. Use precision, recall, the precision-recall curve and lift, not accuracy. Use stratified splits so that every part contains frauds. Look at the number of positive cases, not rows, when judging how much evidence you have. And tie the evaluation to the capacity of the team that will use the results.
Ways to handle imbalance
Four families of techniques exist, and they can be combined.
Resample the training data. Change the mix of rows the model sees.
Undersampling drops some of the majority class. Keep all 200 frauds and a random 2,000 of the genuine transactions, giving a ratio of 10 to 1 instead of 499 to 1. It is fast and discards information.
Oversampling repeats minority rows, so that the 200 frauds each appear many times. It keeps all the data, and a model can memorise the repeated rows.
SMOTE, the Synthetic Minority Over-sampling Technique, creates new minority rows by interpolating between existing ones. Take two fraud cases with features (2, 4) and (4, 8). A synthetic case halfway between them, at a random fraction of 0.5 along the line, is (3, 6). It fills in the minority region without exact copies. It can also create unrealistic cases, particularly with categorical features, and blur the boundary.
Weight the classes. Instead of changing the data, change the cost of errors in the loss function. With 99,800 genuine and 200 fraud cases, a weight of 99,800 ÷ 200 = 499 on each fraud makes the two classes count equally in total. Most libraries offer a classweight setting. It uses all the data and no row is invented.
Move the cutoff. Train as usual, then choose the cutoff from the costs, as Lesson 6.3 showed. A probability model trained on the natural data and thresholded sensibly is often as good as any resampling, as long as the model's probabilities are well ordered.
Use algorithms suited to rarity, such as boosted trees with class weights, or treat the problem as anomaly detection when labelled frauds are very few.
Technique · What it changes · Strength · WeaknessUndersampling · Drops majority rows · Fast, simple · Throws away information, distorts probabilities
Oversampling · Repeats minority rows · Keeps all data · Can memorise duplicates
SMOTE · Adds interpolated minority rows · Fills in the minority region · Can invent unrealistic cases
Class weights · The loss function · No rows changed · Distorts probabilities, needs tuning
Threshold moving · The decision rule · Leaves the model alone · Needs well-ranked probabilities
Anomaly detection · The framing · Works with few or no labels · Hard to tune, flags the merely unusual
A rule that applies to all the resampling methods: do them inside the training folds only. If you oversample or apply SMOTE before splitting, copies and near-copies of the same fraud land in both the training and the validation sets, and the validation score is inflated, a contamination leak of the kind in Lesson 6.4. Never resample the validation or test data. They must keep the real class balance, since the real world will.
Resampling distorts probabilities
Resampling changes the class mix the model sees, so the model's predicted probabilities describe the resampled world and not the real one. If you undersample the genuine transactions so that frauds are one in eleven, the model learns a base rate of about 9 per cent and its probabilities come out far too high.
The correction is exact for undersampling. Suppose you kept a fraction β of the genuine transactions. In the sampled data, the odds of fraud are the true odds divided by β, so the true odds equal β times the sampled odds. Here the frauds are all kept and 2,000 of 99,800 genuine transactions are kept, so β = 2,000 ÷ 99,800 = 0.020. A model trained on this data outputs 0.5 for a transaction, so the sampled odds are 1. The true odds are 0.020 × 1 = 0.020, and the true probability is 0.020 ÷ 1.020 = 0.0196, about 2 per cent. A model score of 0.5 corresponds to a real probability near 2 per cent. A score of 0.8 corresponds to sampled odds of 4, true odds of 0.080 and a probability of 7.4 per cent. A score of 0.2 corresponds to 0.5 per cent.
Predict first: a model trained on undersampled data says a transaction has a 50 per cent chance of being fraud. In production, with the real fraud rate, is the true chance about 50 per cent, about 20 per cent or about 2 per cent? Decide before reading on.
About 2 per cent, as the arithmetic above shows. Treating the output as a real probability would overstate the risk twenty-five-fold. If you only rank cases and review the top, this does not matter, since the ranking is unchanged. But if you use the probability for a cutoff based on costs, or for pricing or provisioning, you must correct it, or calibrate on untouched data as in Lesson 6.3.
Judging a rare-event model
For a model of a rare event, the honest question is what the review team gets for its effort. Suppose a bank can review the top 500 alerts a day. Rank all transactions by the model's score and look at the top. The relevant measures are the precision at k, the share of the top k that are fraud, and the recall at k, the share of all frauds found among them. A random choice has a precision equal to the base rate, 0.2 per cent.
For the 200 frauds a day, suppose the model gives the following.
Reviewed (top k) · Frauds found · Precision at k · Recall at k · Lift over random100 · 25 · 25% · 12.5% · 125 times
500 · 60 · 12% · 30% · 60 times
2,000 · 80 · 4% · 40% · 20 times
The numbers show the usual shape. The first alerts are worth far more than the later ones: the first 100 reviews find 25 frauds, while the next 400 find 35, and the next 1,500 find only 20. With a capacity of 500, the team finds 60 of 200 frauds, 30 per cent, at a precision of 12 per cent. Whether that is good depends on the cost per review and the loss per fraud, which is the capacity logic of Lesson 6.3.
The precision-recall curve shows this trade-off across all depths, and for rare events it is more informative than the ROC curve. The ROC curve measures the false positive rate against a huge number of genuine cases, so a curve that looks excellent can still hide a precision of a few per cent. The baseline of a precision-recall curve is the base rate, and the area under it, called the average precision, is compared with that baseline, not with 0.5.
Anomaly detection: when labels are scarce
Sometimes there are almost no confirmed frauds, or the frauds of tomorrow will look nothing like those of yesterday. Anomaly detection looks for cases that are unusual, without a label. It cannot tell whether an unusual case is bad, only that it is rare, so its output is a list for investigation.
Simple scores
The classic approach is a z-score, the number of standard deviations from the mean, from Lesson 1.3. Because extreme values inflate the mean and the standard deviation, a more robust version uses the median and the median absolute deviation, or MAD, the median of the absolute gaps from the median. The modified z-score is 0.6745 × (value − median) ÷ MAD. If a customer's typical transaction has a median of ₹1,200 with a MAD of ₹300, a transaction of ₹9,000 has a modified z-score of 0.6745 × (9,000 − 1,200) ÷ 300 = 17.5, far beyond the usual flag of about 3.5. These rules judge one feature at a time, and so miss cases that are unusual only in combination, such as a normal amount at an unusual hour in an unusual place.
Isolation forests
An isolation forest takes a clever route. Build many random trees. At each node, pick a feature at random and split at a random value between its minimum and maximum. Keep splitting until every point is alone. An unusual point sits far from the others, so a random split is likely to separate it quickly, and its path from the root to isolation is short. A typical point is surrounded by others, and needs many splits to be isolated, so its path is long. The average path length across the trees is the signal.
The score turns the average path length h into a number between 0 and 1: score = 2 to the power of −h ÷ c(n), where c(n) is the average path length expected for a sample of n points, about 2 × (ln(n − 1) + 0.5772) − 2 × (n − 1) ÷ n. For samples of 256 points, c is 10.24. A point isolated in 3 splits on average scores 2^(−3 ÷ 10.24) = 0.82, a clear anomaly. A point that needs 6 splits scores 0.67, a point at 9 scores 0.54 and a point at 10.24, which is typical, scores 0.5. Scores well above 0.5 flag anomalies, and values near or below 0.5 are ordinary.
Other anomaly methods include one-class support vector machines, distance and density methods such as local outlier factors, and autoencoders, neural networks trained to rebuild normal data and flag cases they rebuild badly. All share the main caveat: a flag means unusual, and most unusual cases are harmless. Combine them with business rules and with whatever labelled cases exist.
Fraud in practice
Fraud has features that make it harder than an ordinary classification problem.
Delayed and incomplete labels. A fraud may only be confirmed weeks later, through a chargeback or a customer complaint, and many frauds are never confirmed. A model trained on recent data sees a partial picture, the censoring problem of Lesson 6.4. Cases that were never reviewed carry no label at all, and if only flagged cases are reviewed, the training data reflects the old model's choices, a feedback loop.
Adversaries who adapt. Fraudsters change tactics when they are caught, so the data drifts by design. Monitoring and frequent refreshes matter more than in most domains, and Lesson 8.2 returns to this.
Features that capture behaviour. The strongest signals are usually velocity features, such as the number of transactions in the last hour or the number of new devices in a day, ratios of the current amount to the customer's recent average, and flags for first-time merchants, countries or devices.
Speed. Many decisions must be made in milliseconds, which limits how complex the model and its features can be.
Cost asymmetry and customer friction. Blocking a genuine customer has a cost beyond the lost sale, so cutoffs are set with explicit costs, often with a step-up check such as an extra authentication in place of an outright block.
A sound workflow is: define the label carefully, with a performance window; build behaviour features available at decision time; split by time with stratification; train a boosted model with class weights; evaluate with precision and recall at the team's capacity, and with the cost at the chosen cutoff; calibrate the probabilities; and add anomaly scores as extra features or as a safety net for new patterns.
python
import numpy as np
from sklearn.ensemble import HistGradientBoostingClassifier, IsolationForest
from sklearn.linearmodel import LogisticRegression
from sklearn.metrics import averageprecisionscore, precisionrecallcurve
from imblearn.oversampling import SMOTE
from imblearn.pipeline import Pipeline as ImbPipeline
class weights: every fraud counts as much as many genuine rows
weights = np.where(ytrain == 1, (ytrain == 0).sum() / (ytrain == 1).sum(), 1.0)
boost = HistGradientBoostingClassifier(learningrate=0.05).fit(Xtrain, ytrain, sampleweight=weights)
SMOTE inside the pipeline, so it is applied to training folds only and never to validation rows
pipe = ImbPipeline([("smote", SMOTE(randomstate=0)), ("clf", LogisticRegression(maxiter=1000))])
pipe.fit(Xtrain, ytrain)
scores = boost.predictproba(Xvalid)[:, 1]
print(averageprecisionscore(yvalid, scores), yvalid.mean()) # compare with the base rate
top = np.argsort(-scores)[:500] # the team can review 500
print(yvalid.iloc[top].mean(), yvalid.iloc[top].sum() / yvalid.sum()) # precision and recall at 500
unsupervised: unusual cases, no labels needed (higher scoresamples means more normal)
iso = IsolationForest(nestimators=300, randomstate=0).fit(Xtrain)
anomalyrank = np.argsort(iso.scoresamples(Xvalid)) # most unusual first
correct a probability from a model trained on undersampled data
beta = 2000 / 99800
def correct(ps):
odds = beta ps / (1 - ps)
return odds / (1 + odds)
Common mistakes
Judging with accuracy or ROC alone. Use precision, recall, the precision-recall curve and lift at the capacity you have.
Resampling before splitting. Copies leak into validation. Resample inside the training folds only.
Resampling the test set. It must keep the real class balance.
Reading undersampled probabilities as real. Correct them, or recalibrate on untouched data.
Ignoring the number of positives. A test set with 30 frauds says little about recall.
Treating an anomaly as a fraud. Unusual is not the same as bad. Route it to review.
Training on reviewed cases only. The old model's choices become the new model's labels. Keep a random review sample.
Forgetting drift. Adversaries adapt, so monitor and refresh.
Key takeaways
Rare outcomes make accuracy useless, starve the loss of signal and make validation noisy. Use precision, recall, precision at k and lift, with stratified, time-based splits.
Handle imbalance by resampling, class weights, a cost-based cutoff, or anomaly detection. A well-ranked model with a good cutoff is often enough.
Resample only inside training folds, and never the validation or test data.
Undersampling distorts probabilities. With 2,000 of 99,800 genuine cases kept, a score of 0.5 means a true probability of about 2 per cent: true odds = kept fraction × sampled odds.
An isolation forest scores unusual points by how quickly random splits isolate them: 2^(−h ÷ c(n)), near 0.5 for ordinary points and near 0.8 for clear anomalies.
Fraud brings delayed labels, feedback loops, adaptive opponents and speed limits. Build behaviour features, evaluate at review capacity, and monitor drift.
Check yourself
1. A day has 50,000 transactions and 50 frauds. What accuracy does a model that flags nothing score, and what recall?
Accuracy 99.9 per cent, recall 0. The accuracy hides the failure.
2. You keep 5 per cent of the majority class and train a model. It outputs 0.6 for a case. What are the sampled odds, and the true odds?
Sampled odds 0.6 ÷ 0.4 = 1.5. True odds 0.05 × 1.5 = 0.075, a probability of 7 per cent.
3. A review team looks at the top 200 alerts and finds 30 frauds, out of 150 frauds in total. What are precision and recall?
Precision 30 ÷ 200 = 15 per cent. Recall 30 ÷ 150 = 20 per cent.
4. Why should SMOTE be applied inside the training fold and not before splitting?
Synthetic and original frauds from the same region would land in both training and validation sets, so the validation score would be inflated.
5. What does an isolation forest score near 0.5 mean, and a score near 0.8?
Near 0.5: an ordinary point that needs a typical number of splits to isolate. Near 0.8: a point isolated unusually quickly, a candidate anomaly.
Try this
Use a dataset with a rare outcome, such as the loan table with a high-risk subset, a public fraud dataset or a simulated one. Set aside forty-five minutes.
1. Record the base rate and the number of positive cases in the training and test sets, using a stratified split. Compute the accuracy of predicting the majority class.
2. Fit a baseline boosted model with no adjustment. Compute average precision and compare it with the base rate. Compute precision and recall at the top 1 per cent and 5 per cent.
3. Refit with class weights, then with undersampling inside a pipeline. Compare the average precision, and compare the raw probabilities with the true rate.
4. Apply the prior correction to the undersampled model and check that its average predicted probability is close to the real base rate. Use the correction simulator for a few scores.
5. Fit an isolation forest without using the labels. See how many of the true positives fall in its top 5 per cent of most unusual cases.
6. Estimate the cost of a review and the value of a caught case, and find the depth at which reviewing stops paying.
7. Write a short note: the method that worked best at your capacity, the precision and recall at that depth, and what you would monitor after launch.
A good result is a table of methods with precision and recall at a fixed review depth, and a calibration check. The next lesson moves from numbers to words: how to turn text into data and measure sentiment.
Tags: Data Science Imbalanced data Fraud detection Anomaly detection SMOTE