Lesson 2.4: Correlation, causation, bootstrapping and A/B tests
Learn to measure how two variables move together, why that is not causation, how to bootstrap an interval, and how to run and read an A/B test.
By the end of this lesson you will be able to measure how strongly two numeric columns move together, explain why a strong link does not show that one causes the other, control for a third variable with a partial correlation, build a confidence interval for almost any statistic with the bootstrap, and plan and read a simple A/B test. You will compute a correlation by hand from five points, see a trend reverse when groups are combined, and work out how many visitors an experiment needs. You should have read Lessons 2.1 to 2.3, since this lesson uses standard errors, t statistics and p-values. The mathematics is arithmetic, one square root, and ideas about fairness of comparison that matter more than any formula.
This lesson closes Module 2. It is the point at which statistics stops being about describing data and starts being about deciding. The commonest business question is not "are these two things related?" but "if we change this, will that improve?", and the gap between those two questions is where most analyses go wrong.
Correlation: how two columns move together
The correlation coefficient, written r, measures how closely two numeric columns follow a straight-line pattern. It runs from −1 to +1. A value of +1 means they rise together on a perfect straight line, −1 means one rises as the other falls on a perfect line, and 0 means there is no straight-line pattern. It does not depend on the units, so you can compare the link between spend and sales with the link between age and income.
To see how it is calculated, take five weeks of advertising spend, in thousands of rupees, and sales, also in thousands: spend of 10, 20, 30, 40 and 50, and sales of 25, 35, 38, 52 and 60.
1. Find each mean: spend averages 30 and sales average 42.
2. Find each deviation from its mean. For spend: −20, −10, 0, 10, 20. For sales: −17, −7, −4, 10, 18.
3. Multiply the two deviations for each week: 340, 70, 0, 100, 360. Their sum is 870.
4. Sum the squared deviations of each column: for spend 400 + 100 + 0 + 100 + 400 = 1,000, and for sales 289 + 49 + 16 + 100 + 324 = 778.
5. Divide: r = 870 ÷ √(1,000 × 778) = 870 ÷ 882.0 = 0.986.
The correlation is 0.986, a very strong positive link. Squaring it gives 0.973, which is the share of the variation in sales that a straight line on spend accounts for. That squared figure returns in the regression lessons. The same quantities also give the slope of the best-fit line: 870 ÷ 1,000 = 0.87, meaning that each additional thousand rupees of spend goes with about 870 rupees more sales in this tiny sample.
Predict first: if you had measured sales in rupees instead of thousands of rupees, what would happen to r? Decide before reading on.
Nothing happens. The correlation is unit-free, because the deviations are divided by the spreads of each column. It also does not change if you swap the two columns, so r says nothing about which one drives the other.
Reading r, and its limits
A rough guide to the size, which depends on the field: values below 0.3 in absolute terms are weak, 0.3 to 0.7 moderate, and above 0.7 strong. Three limits matter more than the labels.
It only sees straight lines. A perfect curve, such as a U shape, can have a correlation near zero. Always draw a scatter plot.
Outliers can fake or hide it. One extreme point can create a strong correlation out of nothing, or destroy one that exists.
A small sample can show a big r by luck. A correlation of 0.8 from five points can easily be chance, and a correlation of 0.1 from a million points is real and tiny.
For the last point, a quick test turns r and the sample size n into a t statistic: t = r × √((n − 2) ÷ (1 − r²)). With r = 0.986 and n = 5 the statistic is 0.986 × √(3 ÷ 0.0277) = 10.3, which is strongly significant even with five points. With r = 0.5 and n = 10 it is 0.5 × √(8 ÷ 0.75) = 1.63, not significant.
Pearson and Spearman
The coefficient above is Pearson's r. A second version, Spearman's rank correlation, replaces each value by its rank (first, second, third...) and computes r on the ranks. It captures any steadily rising or falling pattern, curved or not, and it is much less affected by outliers.
· Pearson · SpearmanMeasures · Straight-line link · Steadily rising or falling link
Uses · The actual values · The ranks
Outliers · Sensitive · Robust
Good for · Roughly linear data without extremes · Skewed data, curves, ordered categories
In pandas both are one line: df["spend"].corr(df["sales"]) for Pearson and .corr(method="spearman") for Spearman.
Correlation is not causation
Ice cream sales and drownings rise together. Neither causes the other: hot weather causes both. The hot weather is a confounder, a third factor linked to both columns that creates a link between them. A correlation between X and Y can arise for four reasons: X causes Y, Y causes X, a third factor causes both, or it is chance. The data alone cannot say which.
Business examples are everywhere. Customers who use a bank's app spend more, but are they spending more because of the app, or are heavy spenders the ones who install it? Branches with more staff have more complaints, but do staff cause complaints, or do busy branches get more staff? Students who attend coaching classes score higher, but who chooses coaching? In each case, acting on the correlation as if it were a cause, say by forcing everyone to install the app, can waste money.
A trend that reverses: Simpson's paradox
Confounding can do something more startling than hide a cause: it can reverse a pattern when groups are combined. A lender compares approval rates at two branches, split by whether the applicant is urban or rural.
Branch A had 200 urban applicants with 160 approved, which is 80 per cent, and 800 rural applicants with 480 approved, which is 60 per cent. In all, 640 of 1,000 were approved, or 64 per cent. Branch B had 800 urban applicants with 560 approved, which is 70 per cent, and 200 rural applicants with 100 approved, which is 50 per cent. In all, 660 of 1,000 were approved, or 66 per cent.
Branch A has the higher approval rate in both groups, 80 against 70 per cent for urban and 60 against 50 per cent for rural. Yet Branch B has the higher overall rate, 66 against 64 per cent. The reason is that Branch A has mostly rural applicants, who are approved less often at both branches, while Branch B has mostly urban ones.
Which comparison is right depends on the question. If you are judging how each branch treats the same kind of applicant, compare within groups, and Branch A is better. If you want to know the total approvals each branch delivers, use the overall figure. The lesson is to look at the groups before accepting an overall number, and to know what mix of cases sits behind it.
Controlling for a third variable: partial correlation
When you suspect a confounder and have measured it, a partial correlation asks how strongly X and Y are linked once the third variable is held fixed. The formula uses the three pairwise correlations:
r(xy.z) = (r(xy) − r(xz) × r(yz)) ÷ √((1 − r(xz)²) × (1 − r(yz)²))
Let ice cream sales be X, drownings Y and temperature Z. Suppose the correlations are 0.80 between sales and drownings, 0.90 between sales and temperature, and 0.85 between drownings and temperature. The numerator is 0.80 − 0.90 × 0.85 = 0.80 − 0.765 = 0.035. The denominator is √((1 − 0.81) × (1 − 0.7225)) = √(0.19 × 0.2775) = √0.0527 = 0.2296. So the partial correlation is 0.035 ÷ 0.2296 = 0.15. Most of the 0.80 link vanishes once temperature is accounted for, which is the signature of a confounder.
Partial correlation only removes what you measured. An unmeasured confounder is still there. This is why the strongest tool for cause is not a clever calculation but a better design: the experiment.
Bootstrapping: an interval for any statistic
The confidence intervals of Lesson 2.3 rest on formulas for the standard error, and those exist only for a few statistics: means, proportions, differences. What is the standard error of a median? Of the 90th percentile? Of a ratio of two averages? The bootstrap answers all of these with one idea: treat your sample as the best available picture of the population, and draw new samples from it.
The procedure is short.
1. Take your sample of n values.
2. Draw n values from it with replacement, meaning a value can be picked more than once, to make a resample the same size as the original.
3. Compute your statistic, such as the median, on the resample.
4. Repeat 5,000 or 10,000 times.
5. The spread of those statistics shows how much the statistic varies. The middle 95 per cent of them, from the 2.5th to the 97.5th percentile, is a 95 per cent bootstrap interval.
Consider eight order values in rupees: 120, 150, 90, 400, 110, 130, 95 and 105. The mean is 1,200 ÷ 8 = 150, and the median is 115. A formula-based interval for the mean assumes a bell-shaped sampling distribution, which is doubtful with eight values and one of 400. The bootstrap makes no such assumption. Resamples that include the 400 several times will have a large mean, and those that skip it a small one, so the bootstrap distribution is wide and skewed to the right, which tells you something real about the data that a symmetric formula would hide.
python
import numpy as np
rng = np.random.defaultrng(1)
orders = np.array([120, 150, 90, 400, 110, 130, 95, 105])
medians = [np.median(rng.choice(orders, size=len(orders), replace=True))
for in range(10000)]
low, high = np.percentile(medians, [2.5, 97.5]) # 95% bootstrap interval for the median
· Formula-based interval · Bootstrap intervalWorks for · Means, proportions, simple differences · Almost any statistic
Needs · Assumptions about shape or a known formula · A reasonably representative sample
Effort · Immediate · Needs a computer, easy to run
Weak when · Data is skewed and the sample small · The sample is tiny or unrepresentative
The bootstrap has one hard limit: it can only reflect the variation that is in your sample. If the sample is biased, or too small to include rare cases, the bootstrap will repeat that flaw faithfully.
Experiments and A/B tests
The cleanest way to learn whether X causes Y is to randomise: assign people at random to get the change or not, and compare the results. Random assignment breaks any link between who gets the treatment and every other factor, measured or not, so a difference afterwards can be put down to the treatment, plus chance. This is a randomised controlled trial, and in business it is called an A/B test: group A (control) sees the current version, group B (variant) sees the new one.
A sound A/B test follows a sequence.
1. State the hypothesis and the one primary metric before you start, such as conversion rate from visit to sign-up.
2. Choose the smallest effect worth detecting, since you cannot detect everything.
3. Work out the sample size needed, so the test has enough power.
4. Randomise at the level of the unit you care about, usually the user, not the visit.
5. Run it for the planned time and sample, without stopping early.
6. Analyse with a test for two proportions, report the effect and its interval, and check guardrail metrics.
Reading a result
A bank tests a new sign-up screen. Control converts 400 of 5,000 visitors, 8.0 per cent. The variant converts 440 of 5,000, 8.8 per cent. The variant looks 10 per cent better. Is it?
The pooled rate is (400 + 440) ÷ 10,000 = 8.4 per cent. The standard error of the difference is √(0.084 × 0.916 × (1 ÷ 5,000 + 1 ÷ 5,000)) = √(0.07694 × 0.0004) = 0.00555, or 0.55 percentage points. The difference is 0.8 points, so z = 0.8 ÷ 0.555 = 1.44. The p-value is about 0.15. The result is not significant. The gain may be real, but a test of this size cannot tell it from chance.
How many users do you need?
The sample size for a two-proportion test depends on the base rate, the lift you want to detect, the significance level and the power you want. For 5 per cent significance and 80 per cent power, the standard calculation is
n per group = 7.84 × (p1 × (1 − p1) + p2 × (1 − p2)) ÷ (p2 − p1)²
where 7.84 is (1.96 + 0.84)², the squared sum of the multipliers for significance and power. For a lift from 8.0 to 8.8 per cent: p1 × (1 − p1) = 0.0736, p2 × (1 − p2) = 0.0803, their sum is 0.1539, and (0.008)² = 0.000064. So n = 7.84 × 0.1539 ÷ 0.000064 = 18,850 per group, about 37,700 users in all. The test of 5,000 per group was far too small for a lift of that size, which explains the non-significant result.
A sample size calculation forces the useful conversation before the test: what is the smallest improvement that would change a decision? If a 0.8 point lift is not worth acting on, a much smaller test will do. If you want to detect a 0.2 point lift, you will need millions of visitors, and it may not be worth testing at all.
Peeking, and other ways tests go wrong
The most common failure is peeking: checking the result every day and stopping when it looks significant. Each look is another chance for a false positive, so the overall false positive rate rises well above 5 per cent. With five planned looks at the data and a stop at the first significant one, the true rate is about 14 per cent, and with ten looks about 19 per cent.
Other traps deserve a place on your checklist.
Sample ratio mismatch: if you planned a 50:50 split and got 52:48 with thousands of users, the assignment is probably broken, and results are not trustworthy.
Novelty effects: users click on anything new at first. A test that runs only a few days may overstate a lasting benefit.
Seasonality and calendar: run for whole weeks, so that weekdays and weekends are balanced.
Interference: if users affect each other, as in marketplaces or referral schemes, one group's treatment leaks into the other.
Many metrics: if you track twenty outcomes, one will move by chance. Name one primary metric in advance, and treat the rest as secondary or guardrails.
Guardrail metrics: check that a gain in conversion did not come with a rise in complaints or defaults.
Observational data when you cannot run a test
Often you cannot randomise: you cannot assign borrowers to a recession. Then you work with observational data, compare similar units, control for known confounders, and look for natural experiments, such as a policy that changed at a date or a threshold that placed similar people on either side of a line. These methods, covered in Lesson 7.4, can strengthen a causal claim but never make it as certain as a randomised test. When you report such a result, say that it is an association adjusted for the factors you measured.
Common mistakes
Reading a correlation as a cause. Look for confounders, reverse causation and chance.
Reporting r without a scatter plot. Curves, clusters and outliers can fool the number.
Ignoring group structure. An overall pattern can reverse inside the groups, as in Simpson's paradox.
Controlling for the wrong thing. Controlling for a variable that lies on the path between cause and effect can hide the real effect.
Bootstrapping a biased or tiny sample. The bootstrap reproduces the sample's flaws.
Launching an A/B test without a sample size. Underpowered tests produce noise and false confidence.
Peeking and stopping early. Fix the sample size and the stopping rule in advance.
Testing many metrics and reporting the winner. Declare one primary metric before you start.
Key takeaways
Correlation r runs from −1 to +1, is unit-free and sees only straight-line links. Always plot the data, and use Spearman's rank version for skewed data or curves.
Correlation does not show cause. A confounder, reverse causation or chance can produce the same pattern, and Simpson's paradox shows a trend reversing when groups are combined.
A partial correlation removes the part of a link explained by a measured third variable, but not by an unmeasured one.
The bootstrap resamples your data with replacement to give an interval for almost any statistic, without a formula, and it repeats whatever flaws the sample has.
Randomisation is the strongest route to cause. An A/B test needs a pre-stated primary metric, a sample size, proper randomisation and a fixed stopping rule.
A lift from 8.0 to 8.8 per cent needs about 18,850 users per group. Peeking five times pushes the false positive rate from 5 to about 14 per cent.
Check yourself
1. Spend and sales have r = 0.9. If you change sales from thousands to rupees, what happens to r?
Nothing. The correlation is unit-free.
2. Two columns have a perfect U-shaped relationship. Could r be close to zero?
Yes. Pearson's r only measures straight-line links, so draw a scatter plot.
3. Sales and drownings correlate at 0.7, both correlate with temperature at 0.9 and 0.8. What does the partial correlation suggest?
The numerator is 0.7 − 0.72 = −0.02, so the partial correlation is near zero. Temperature explains the link.
4. Why does the bootstrap resample with replacement?
So each resample is the same size as the original yet differs from it, mimicking the variation between samples drawn from the population.
5. A test of 3,000 users per group shows a conversion lift from 5 to 5.3 per cent. Is it likely to be significant, and why does it matter?
The z statistic is only about 0.5, so no. A test of that size is underpowered for a 0.3 point lift.
Try this
Use a real or made-up dataset with at least two numeric columns and a yes-or-no outcome. Set aside forty-five minutes.
1. Compute the Pearson and Spearman correlation of two numeric columns and draw the scatter plot. Say whether the plot supports the number.
2. Name two possible confounders. If you have one in the data, compute the partial correlation, or split the data by that variable and look for a reversed pattern.
3. Take a statistic with no easy formula, such as the median or the 90th percentile of a column, and compute a 95 per cent bootstrap interval with 5,000 resamples. Compare its width with the interval for the mean.
4. Imagine a change you could test, such as a new message to customers. Write the primary metric, the smallest lift worth detecting, and use the A/B simulator to find how many users per group you need.
5. If you have two groups with yes-or-no outcomes, compute the difference, its standard error and the z statistic. Decide whether the test was large enough.
6. Write a one-page note: what moves with what, what could confound it, how you would test cause, and how many users the test needs.
A good result is a note that separates what the data shows from what you suspect, and says exactly what test would settle it. With that, Module 2 is complete. You can now describe data, reason about chance, and judge whether a difference is real. Module 3 turns to the methods analysts use every day to answer business questions with real tables.
Tags: Data Science Correlation Bootstrap A/B testing Causation