Lesson 2.1: Probability and distributions, from coins to bell curves
Learn the rules of probability, how Bayes' rule corrects a gut feeling, and how the binomial and normal distributions describe real business data.
By the end of this lesson you will be able to calculate the chance of simple and combined events, use Bayes' rule to turn a test result into a sensible probability, and describe a column of data with the binomial, normal and a few other distributions. You will see why a fraud alert that is right 90 per cent of the time can still be wrong more than 8 times in 10, how a small default rate compounds across a loan book, and how a bell curve turns a credit score into a share of applicants. You should have read Module 1, since distributions are the mathematical version of the histograms and summaries you met there. The mathematics is arithmetic, powers and one square root, and each idea is worked by hand before it is used on a large example.
Probability is the language that lets us say how sure we are. Every later lesson in this course leans on it: a confidence interval is a statement about probability, a p-value is a probability, and a model that predicts default gives a probability. If you are shaky here, the rest of the course will feel like recipes. If you are solid, it will feel like consequences.
What a probability means
A probability is a number between 0 and 1 that says how likely an event is. Zero means it cannot happen and 1 means it must. Often we write it as a percentage. The most useful way to think about it for data work is as a long-run frequency: if a borrower has a 4 per cent probability of default, then among a very large number of similar borrowers about 4 in every 100 will default. It does not say which ones.
Three rules do most of the work.
The complement rule. The chance that an event does not happen is 1 minus the chance that it does. If the default probability is 0.04, the probability of repaying is 0.96.
The addition rule for either of two exclusive events. If two events cannot both happen, the chance that one or the other happens is the sum. A customer cannot be in both the "salaried" and "self-employed" group, so P(salaried or self-employed) = P(salaried) + P(self-employed).
The multiplication rule for both of two independent events. If one event has no influence on the other, the chance that both happen is the product. If two unrelated loans each default with probability 0.04, the chance that both default is 0.04 × 0.04 = 0.0016, or 0.16 per cent.
Independence is the word to watch. Two loans to borrowers in the same town hit by the same flood are not independent, and the multiplication rule would understate the chance that they default together. A good deal of financial risk is exactly this: events that were assumed independent and were not.
Predict first: a lender makes three loans to unrelated borrowers, each with a 4 per cent chance of default. What is the chance that at least one defaults? Make a guess before you read on. Many people answer "about 12 per cent", which is 3 × 4.
The answer is 11.5 per cent, close to the guess but not equal to it. The trick is to calculate the complement. The chance that none defaults is 0.96 × 0.96 × 0.96 = 0.8847. So the chance of at least one default is 1 − 0.8847 = 0.1153. The simple sum of 12 per cent is slightly too high because it counts the rare case where two or three loans default more than once. For small probabilities and few loans the sum is a fine shortcut. As the number of loans grows, the sum goes badly wrong, because it can exceed 100 per cent. Try it with the sliders.
With a 4 per cent default rate, the chance of at least one default is already 11.5 per cent for three loans, 33.5 per cent for ten and 87 per cent for fifty. A lender with a large book is not asking whether any loan will default. It is asking how many.
Conditional probability and Bayes' rule
The probability of an event often changes once you know something else. The conditional probability of A given B, written P(A | B), is the chance of A among the cases where B is true. The chance a customer defaults is different for those who have missed an earlier payment, so P(default | missed payment) is not the same as P(default).
The two directions of a conditional probability are not the same thing, and mixing them up is one of the most expensive mistakes in applied statistics. P(positive test | fraud) is how good the test is at catching fraud. P(fraud | positive test) is what a person reviewing an alert needs to know. Bayes' rule converts one into the other. In words: the probability of the cause given the evidence equals the probability of the evidence given the cause, times the prior probability of the cause, divided by the overall probability of the evidence.
A worked example shows why it matters. A bank's fraud model has these properties. About 1 per cent of transactions are fraudulent. The model flags 90 per cent of fraudulent transactions. It also wrongly flags 5 per cent of genuine ones. An alert comes in. What is the chance it is genuine fraud?
Imagine 10,000 transactions. One per cent, or 100, are fraudulent, and the model catches 90 of them and misses 10. The other 9,900 are genuine, and the model wrongly flags 5 per cent of them, which is 495. So the model flags 90 + 495 = 585 transactions, and only 90 of them are fraud. The chance that a flagged transaction is fraud is 90 ÷ 585 = 15.4 per cent.
A model that is right 90 per cent of the time on fraud still produces alerts that are wrong about 85 per cent of the time, because genuine transactions vastly outnumber fraudulent ones. This effect is called the base rate fallacy: ignoring how rare the thing is in the first place. It is the reason that screening for rare diseases, flagging terrorist financing and detecting fraud all face a mountain of false alarms, and the reason a good analyst asks "how common is it?" before praising an accuracy figure.
Raise the base rate to 20 per cent and the share of real alerts rises to 81.8 per cent. Drop the false-alarm rate to 1 per cent and the figure at a 1 per cent base rate rises to 47.6 per cent. Two levers, how common the event is and how often the test cries wolf, matter more than the headline accuracy.
Random variables, expected value and spread
A random variable is a number whose value depends on chance, such as the number of defaults in a month or the amount of a claim. A probability distribution says which values it can take and how likely each is. The expected value is the long-run average, found by multiplying each value by its probability and adding. It need not be a value you can actually observe. A fair die has an expected value of 3.5.
Expected value is how lenders think about loss. If a loan of ₹10 lakh has a 3 per cent chance of default, and a lender would recover 40 per cent of it after a default so that the loss given default is 60 per cent, the expected loss is 0.03 × 0.60 × ₹10 lakh = ₹18,000. No single loan loses ₹18,000. Most lose nothing and a few lose ₹6 lakh, and the lender prices the interest rate so that the expected loss is covered. The variance and standard deviation you met in Lesson 1.3 apply here too: they measure how far outcomes typically fall from the expected value, and so how much capital is needed as a cushion.
A second use of expected value: is a promotion worth it?
Expected value is not only for losses. A retailer plans a ₹150 cashback offer. Based on past campaigns, 12 per cent of customers who receive it will redeem it, and a redeemed purchase earns the retailer ₹900 in margin. A quick calculation gives an expected gain per customer of 0.12 × (900 − 150) = ₹90. That sounds excellent, but it hides a question about probability: how many of those redeemers would have bought anyway? Suppose 40 per cent of them would have, so the offer only creates new margin for the other 60 per cent. The cashback is still paid to everyone who redeems, so the expected gain per customer falls to 0.12 × (0.6 × 900 − 150) = 0.12 × 390 = ₹46.80. The offer is still positive, but by half. This is a first glimpse of a theme that returns in Lesson 2.4 and in the causal inference lesson: the number that matters is the difference the action makes, not the outcome that follows it.
The binomial distribution: counting successes
When you repeat the same yes-or-no trial a fixed number of times, and each trial has the same probability of success and is independent of the others, the number of successes follows a binomial distribution. It has two parameters: the number of trials n and the probability p of success in each. The chance of exactly k successes is C(n, k) × p^k × (1 − p)^(n − k), where C(n, k) counts the ways to choose which k trials succeed. Its mean is n × p and its standard deviation is √(n × p × (1 − p)).
Take a lender with 20 loans, each with a 4 per cent default chance. The expected number of defaults is 20 × 0.04 = 0.8. The chance of none is 0.96 to the power 20, which is 0.442, or 44.2 per cent. The chance of exactly one is 20 × 0.04 × 0.96 to the power 19, which is 36.8 per cent. For two, three and four defaults the values are 14.6, 3.6 and 0.6 per cent.
Notice the shape. The mean of 0.8 sits between zero and one, and the distribution is lopsided, with its peak at zero and a thin tail to the right. With few loans and a small probability, the binomial is skewed. With 500 loans at the same 4 per cent, the mean is 20, the standard deviation is √(500 × 0.04 × 0.96) = 4.4, and the shape becomes almost symmetric and bell-like. This is a first hint of the central limit idea, which Lesson 2.2 develops.
In Python the calculation is one line with the scipy.stats library.
python
from scipy import stats
stats.binom.pmf(0, n=20, p=0.04) # chance of exactly 0 defaults: 0.442
stats.binom.pmf(1, n=20, p=0.04) # exactly 1: 0.368
1 - stats.binom.cdf(4, n=20, p=0.04) # chance of 5 or more defaults
stats.binom.mean(20, 0.04), stats.binom.std(20, 0.04)
The normal distribution: the bell curve
The most famous continuous distribution is the normal distribution, the bell curve. It is described by two numbers, its mean and its standard deviation, and it is symmetric around the mean. Many measurements are roughly normal because they are the sum of many small independent influences: heights, errors in a measuring device, and the average of a large sample. You met the empirical rule in Lesson 1.3: about 68 per cent of values lie within one standard deviation of the mean, 95 per cent within two and 99.7 per cent within three.
To find the share of a normal distribution above or below a point, convert to a z-score and read the share from a table or a function. A z-score of 0 splits the data 50:50, a z-score of 1 leaves 15.9 per cent above, a z-score of 1.64 leaves 5 per cent above and a z-score of 2 leaves 2.3 per cent above.
Suppose applicants' credit scores are roughly normal with a mean of 650 and a standard deviation of 50. A lender wants to approve only those above 700. The cutoff is one standard deviation above the mean, so z = (700 − 650) ÷ 50 = 1, and about 15.9 per cent of applicants qualify. Raise the cutoff to 750, which is z = 2, and only 2.3 per cent qualify. Try changing the cutoff.
The normal distribution is also the basis of many later tools, which is why you should be comfortable converting between a value, a z-score and a share. But be careful. Many business columns are not normal. Incomes, claim sizes and transaction amounts are right-skewed, and applying normal-curve shares to them gives wrong answers, usually by understating the chance of extreme values. Check the histogram before you use the bell curve.
A small family of distributions
A handful of distributions cover most situations. The table shows what each describes, so that you can recognise the situation before reaching for a formula.
Distribution · Describes · Example · ShapeBernoulli · One yes-or-no trial · Does this loan default? · Two bars
Binomial · Number of successes in n trials · Defaults among 20 loans · Skewed for small p, bell-like for large n
Poisson · Count of events in a fixed time or space · Calls to a help desk per hour · Skewed, mean equals variance
Uniform · All values in a range equally likely · A random delay between 0 and 10 minutes · Flat
Normal · Sums and averages, many measurements · Credit scores, measurement error · Symmetric bell
Exponential · Waiting time between events · Minutes between customer arrivals · Steeply falling
Log-normal · Positive, right-skewed quantities · Incomes, claim sizes · Long right tail
The Poisson distribution deserves a note, because counting events is so common. If a help desk receives on average 4.2 calls an hour, the number of calls in an hour is roughly Poisson with mean 4.2, and so is its variance. The chance of no calls in an hour is e to the power −4.2, which is 1.5 per cent. The exponential distribution describes the waiting time between those calls, with a mean of 1 ÷ 4.2 hours, or about 14 minutes.
Common mistakes
Adding probabilities that should be combined. For "at least one", use the complement. For "both", multiply only if the events are independent.
Assuming independence. Borrowers in one town, transactions by one stolen card and defaults in one recession are linked. Independence is a claim, not a default.
Reading P(A | B) as P(B | A). A test that catches 90 per cent of fraud does not mean 90 per cent of alerts are fraud.
Ignoring the base rate. A rare event with a good-but-imperfect test produces mostly false alarms.
Using the normal curve for skewed data. Look at the histogram first. Income, claims and delays often need a log-normal or an empirical approach.
Treating the expected value as a prediction. The expected number of defaults is 0.8 in the 20-loan example, but you will see 0 or 1, never 0.8.
Key takeaways
A probability is a long-run frequency between 0 and 1. Use the complement for "at least one", and multiply only independent events.
Conditional probability changes with what you know, and Bayes' rule turns P(evidence | cause) into P(cause | evidence).
The base rate matters. A test that is right 90 per cent of the time on a 1 per cent event produces mostly false alarms.
The expected value is the long-run average, and the standard deviation measures how much outcomes vary around it.
The binomial distribution counts successes in a fixed number of independent trials, with mean n × p. The normal distribution describes sums and averages, and z-scores convert any value to a share.
Pick the distribution to match the situation: counts, waiting times, positive skewed amounts and symmetric measurements each have a natural one.
Check yourself
1. Two independent loans each default with probability 0.05. What is the chance that at least one defaults?
1 − 0.95 × 0.95 = 1 − 0.9025 = 0.0975, or 9.75 per cent.
2. A test catches 80 per cent of cases of a condition that affects 2 per cent of people, and wrongly flags 10 per cent of the rest. Of 10,000 people, how many are flagged and how many of those are real?
Real and flagged: 160 of 200. False alarms: 980 of 9,800. Flagged in all: 1,140. Real among flagged: 160 ÷ 1,140 = 14 per cent.
3. A loan of ₹5 lakh has a 2 per cent chance of default and a loss of 50 per cent if it defaults. What is the expected loss?
0.02 × 0.5 × ₹5 lakh = ₹5,000.
4. A lender has 500 loans at a 4 per cent default rate. What are the expected number of defaults and the standard deviation?
Expected: 500 × 0.04 = 20. Standard deviation: √(500 × 0.04 × 0.96) = √19.2 = 4.4.
5. Scores are roughly normal with mean 650 and standard deviation 50. Roughly what share score above 750?
The z-score is 2, so about 2.3 per cent score above 750.
Try this
Use your own data or a small made-up portfolio. Set aside forty minutes.
1. Choose a yes-or-no outcome you can count: a late payment, an order returned, an email opened. Estimate its probability from at least 100 rows.
2. Use the complement rule to work out the chance that at least one of ten such events happens, assuming independence. Check it with the first simulator. Then write one sentence on whether independence is realistic here.
3. Find a test, alert or rule in your work that flags cases, such as a fraud rule or a spam filter. Estimate how common the thing is, how many real cases it catches and how many genuine cases it wrongly flags. Use the Bayes simulator to find the share of alerts that are real.
4. For a numeric column, draw a histogram and decide whether a bell curve is a fair description. If it is, find the share above a chosen value using a z-score. If it is skewed, say what you would use instead.
5. Compute the expected value of something you care about, such as the expected cost of returns per order, and name one reason the real outcome will differ from it.
6. Write a half-page note describing what you found, including one conclusion that changed once you accounted for the base rate or for independence.
A good result is a note that states each probability, shows the calculation and names its assumption. The next lesson moves from describing chance to using samples: how a few thousand rows can tell us something reliable about millions, and how that goes wrong.
Tags: Data Science Probability Distributions Bayes rule Statistics