Lesson 2.3: Confidence intervals, t-tests and hypothesis tests
Learn to put an honest margin around an estimate, run a t-test and a chi-square test by hand, read a p-value correctly and avoid false positives.
By the end of this lesson you will be able to attach a margin of uncertainty to an average or a rate, decide whether a difference between two groups is likely to be real or just chance, and read a p-value without falling for the usual traps. You will compute a confidence interval, a one-sample t-test, a two-sample t-test, a paired test and a chi-square test with small numbers you can check by hand, and see why running many tests produces false findings. You should have read Lesson 2.2, since a confidence interval is built from a standard error, and Lesson 2.1 for z-scores and the normal curve. The mathematics is arithmetic and square roots, and the Python needed is a few lines.
Business questions are rarely "what is the average?". They are "is it better than before?", "is branch A really slower than branch B?" and "did the new screen change conversion?". A raw difference is not an answer, because samples differ by chance. The tools in this lesson turn a difference into a statement about how surprising it would be if nothing had really changed.
From a standard error to a confidence interval
A single number such as "the average repayment takes 42 days" hides its own uncertainty. A confidence interval adds it back. It is a range around the estimate, built so that if you repeated the whole study many times, a stated share of the ranges, usually 95 per cent, would contain the true value. The recipe is the estimate plus and minus a multiplier times the standard error.
Take a sample of 64 repayments with an average of 42 days and a standard deviation of 16. The standard error is 16 ÷ √64 = 2. For 95 per cent confidence the multiplier from the normal curve is 1.96, so the margin is 1.96 × 2 = 3.92 days. The interval runs from 42 − 3.92 = 38.1 to 42 + 3.92 = 45.9 days. In words: we estimate the true average at 42 days, give or take about 4.
Predict first: if you want to be 99 per cent confident instead of 95 per cent, will the interval become wider or narrower? Decide before you read on.
It becomes wider. More confidence needs a bigger net. The multiplier for 99 per cent is 2.576 and for 90 per cent it is 1.645. With the same data the 99 per cent interval is 42 ± 5.15, from 36.8 to 47.2 days. There is always a trade-off between how sure you are and how precise the statement is. A very narrow interval can only be bought with more data, because the standard error shrinks with the square root of n.
Small samples and the t distribution
The multiplier of 1.96 assumes we know the true standard deviation. In practice we estimate it from the sample, which adds extra uncertainty, especially when the sample is small. The t distribution corrects for that. It looks like the normal curve but with heavier tails, and its shape depends on the degrees of freedom, which for a mean is n − 1. The smaller the sample, the larger the multiplier.
Repeat the repayment example with only 16 repayments, the same mean of 42 and standard deviation of 16. The standard error is 16 ÷ √16 = 4, the multiplier for 15 degrees of freedom is 2.131, and the margin is 8.5 days, giving 33.5 to 50.5. Using 1.96 would have given a margin of 7.8, which claims more precision than the data justifies. Software uses the t distribution automatically, so you rarely compute it by hand, but you should know why small samples give wider intervals.
Intervals for proportions
For a rate, such as a conversion rate or a default rate, the standard error is √(p × (1 − p) ÷ n). If 80 of 1,000 visitors converted, p is 0.08, the standard error is √(0.08 × 0.92 ÷ 1,000) = 0.0086, the margin is 1.96 × 0.0086 = 0.0168, and the interval is 6.3 to 9.7 per cent. So "8 per cent" is better written as "about 8 per cent, plausibly between 6.3 and 9.7".
What a confidence interval means, and what it does not
The correct reading is about the method: if we repeated the sampling many times and built an interval each time, about 95 per cent of the intervals would contain the true value. It is tempting to say "there is a 95 per cent chance the true value is in this interval", and in everyday talk people do. Strictly, the true value is fixed and it either is in this particular interval or is not. The 95 per cent describes how reliable the procedure is. For most business purposes the practical reading is fine: here is a range of values the data cannot rule out.
Two more points matter. A confidence interval says nothing about bias: a biased sample gives a precise interval around the wrong number. And intervals that overlap do not automatically mean that two groups are not different, because the interval for the difference is a separate calculation.
The logic of a hypothesis test
A hypothesis test asks whether the data is surprising if nothing special is going on. It has a fixed sequence.
1. State the null hypothesis, the "nothing is going on" claim, such as "the new process has the same average time as the old one".
2. State the alternative hypothesis, such as "the new process time is different".
3. Compute a test statistic, a number that measures how far the data is from what the null predicts, in standard errors.
4. Find the p-value: the probability, if the null were true, of seeing a result at least this extreme.
5. Compare with a pre-chosen significance level, usually 5 per cent. If the p-value is below it, call the result statistically significant and reject the null.
The p-value is the most misread number in statistics. It is not the probability that the null is true. It is not the probability that the result is due to chance. It is the probability of data this extreme, assuming the null is true. A small p-value says the data is hard to reconcile with the null. A large p-value says only that the data does not provide strong evidence against it, which is not the same as evidence that the null is true.
There are two ways to be wrong. A Type I error, or false positive, is rejecting a true null. Its probability is the significance level, so at 5 per cent you accept a one in twenty chance of a false alarm when nothing is happening. A Type II error, or false negative, is failing to reject a false null. The chance of avoiding it is the test's power, and power rises with the sample size and with the size of the real effect. A test with low power usually fails to detect real effects, and the "not significant" result tells you little.
· The null is true (no real effect) · The null is false (a real effect)Test says significant · Type I error (false positive) · Correct detection (power)
Test says not significant · Correct · Type II error (false negative)
## The one-sample t-test
A one-sample t-test compares a sample average with a fixed claim. A bank says that its loan processing takes 3 days on average. An audit of 25 loans finds an average of 3.4 days with a standard deviation of 1.0. Is the claim wrong?
The null is that the true average is 3 days. The standard error is 1.0 ÷ √25 = 0.2. The test statistic is t = (3.4 − 3.0) ÷ 0.2 = 2.0, which says the sample average is two standard errors from the claim. With 24 degrees of freedom the critical value for 5 per cent significance in a two-sided test is 2.064, and the p-value is about 0.057. The result just misses the 5 per cent threshold. The honest conclusion is not "the claim is true" but "the data is suggestive, and a larger audit would settle it".
The two-sample t-test
Often the question compares two groups. Branch A resolved 40 cases with a mean of 5.2 days and a standard deviation of 2.0. Branch B resolved 50 cases with a mean of 4.4 days and a standard deviation of 2.4. The gap is 0.8 days. Is it real?
The standard error of the difference is √(2.0² ÷ 40 + 2.4² ÷ 50) = √(0.100 + 0.115) = √0.215 = 0.464. The test statistic is t = 0.8 ÷ 0.464 = 1.72. The p-value is about 0.09, above 5 per cent. A gap of 0.8 days looks meaningful on the shop floor, and the data is not strong enough to call it more than chance. This version, which does not assume the two groups have equal spreads, is called Welch's t-test, and it is the safe default.
Increase both group sizes to 200 and the same gap of 0.8 gives a standard error of 0.22 and a t statistic of 3.6, clearly significant. Nothing about the branches has changed. The data is simply stronger.
The paired t-test
When the same people are measured twice, such as before and after training, a paired test is more powerful. Instead of comparing two groups, you compute each person's difference and test whether the average difference is zero. If 36 agents improved by an average of 1.5 resolved cases a day, with a standard deviation of the differences of 3, then the standard error is 3 ÷ √36 = 0.5 and t = 1.5 ÷ 0.5 = 3.0. The p-value is about 0.005, so the improvement is very unlikely to be chance. Pairing removes the differences between agents, which is why it can detect effects that a two-group comparison would miss.
The chi-square test for categories
For counts in categories, the tool is the chi-square test of independence. It asks whether two categorical variables are linked. A lender compares approval by channel. Of 300 app applications, 210 were approved, which is 70 per cent. Of 200 branch applications, 120 were approved, which is 60 per cent. Is the difference real?
If channel did not matter, the overall approval rate of 330 ÷ 500 = 66 per cent would apply to both. The expected approvals would be 0.66 × 300 = 198 for the app and 0.66 × 200 = 132 for the branch, and the expected rejections would be 102 and 68. For each of the four cells, compute (observed − expected)² ÷ expected: for app approvals (210 − 198)² ÷ 198 = 0.727, for branch approvals (120 − 132)² ÷ 132 = 1.091, for app rejections (90 − 102)² ÷ 102 = 1.412, for branch rejections (80 − 68)² ÷ 68 = 2.118. The total is 5.35. With one degree of freedom, the 5 per cent critical value is 3.84, and the p-value is about 0.02. The app channel has a genuinely higher approval rate in this sample, which does not tell us why: app applicants may simply be different people.
Choosing the test
Question · Data · Test · ExampleIs one average equal to a claimed value? · One numeric column · One-sample t-test · Is processing time 3 days?
Do two independent groups differ in average? · Numeric by two groups · Welch's two-sample t-test · Branch A against branch B
Did the same units change? · Before and after pairs · Paired t-test · Agents before and after training
Do two rates differ? · Yes-or-no outcome by two groups · Two-proportion z-test or chi-square · Conversion on old and new page
Are two categories linked? · Counts in a table · Chi-square test of independence · Approval by channel
Do three or more groups differ in average? · Numeric by several groups · One-way ANOVA · Resolution time across five branches
The first three compare means, and the last three handle rates, counts and several groups. These tests assume the data is roughly bell-shaped, or the sample is large enough for the central limit idea to apply, and that observations are independent. For badly skewed data or small samples, rank-based tests or the bootstrap in Lesson 2.4 are safer.
In Python, each test is a single call to the scipy.stats library.
python
from scipy import stats
stats.ttest1samp(times, popmean=3) # one-sample t-test
stats.ttestind(a, b, equalvar=False) # Welch's two-sample t-test
stats.ttestrel(before, after) # paired t-test
table = [[210, 90], [120, 80]] # approved, rejected by channel
chi2, p, dof, expected = stats.chi2contingency(table, correction=False)
Each call returns a test statistic and a p-value. Read the p-value as a measure of surprise, and then look at the size of the difference.
Significance is not importance
A tiny difference can be statistically significant if the sample is huge. With a million customers, a gain of 0.02 percentage points in conversion may be highly significant and commercially worthless. The reverse also holds: a large effect in a small sample may fail to reach significance. Always report an effect size, the size of the difference in useful units, together with its confidence interval. One standard measure is Cohen's d, the difference in means divided by the pooled standard deviation. In the branch example the pooled standard deviation is about 2.23, so d = 0.8 ÷ 2.23 = 0.36, a small to moderate effect. Roughly, 0.2 is small, 0.5 is medium and 0.8 is large.
· Statistical significance · Practical importanceAsks · Is this difference unlikely to be chance? · Is this difference big enough to matter?
Depends on · Sample size and spread · Cost, benefit and context
Reported as · A p-value or interval · An effect size in rupees, days or points
## The multiple testing trap
If you run one test at the 5 per cent level on data with no real effect, there is a 5 per cent chance of a false positive. If you run 20 such tests, the chance that at least one is falsely significant is 1 − 0.95 to the power 20, which is 64 per cent. The chance rises quickly with the number of tests, as the chart shows.
This is how a team slicing a result by region, age group, product and week finds "a significant effect" in some corner of the data, even when there is none. It is sometimes called p-hacking, and it can happen innocently. Defences include deciding the hypotheses before looking at the data, counting how many comparisons you made, and adjusting for them. The simplest adjustment is the Bonferroni correction: divide the significance level by the number of tests, so 20 tests at an overall 5 per cent level need each test to reach 0.25 per cent.
Common mistakes
Reading the p-value as the chance the null is true. It is the chance of data this extreme if the null were true.
Treating "not significant" as "no effect". A small sample may simply lack the power to detect a real effect.
Treating "significant" as "important". Check the size of the effect and its interval.
Running many tests and reporting the one that worked. Count the comparisons and correct for them.
Using a two-sample test for before-and-after on the same people. Use a paired test.
Stopping when the p-value dips below 0.05. Decide the sample size in advance. Lesson 2.4 shows why peeking inflates false positives.
Forgetting the assumptions. Independence, reasonable shape and no bias matter more than the arithmetic.
Using the normal multiplier for a tiny sample. Use the t distribution.
Key takeaways
A confidence interval is the estimate plus and minus a multiplier times the standard error. A higher confidence level makes it wider, and more data makes it narrower.
Small samples need the t distribution, whose multiplier is larger and falls towards 1.96 as the sample grows.
A hypothesis test asks how surprising the data is if the null is true. The p-value is that surprise, not the probability that the null is true.
Type I errors are false alarms at the chosen significance level. Type II errors are missed effects, and power measures the ability to avoid them.
Match the test to the question: t-tests for means, paired tests for before and after, chi-square for categories, ANOVA for several groups.
Report effect sizes and intervals, not just p-values, and correct for the number of tests you run.
Check yourself
1. A sample of 100 has a mean of 50 and a standard deviation of 10. What is the 95 per cent confidence interval?
The standard error is 1, the margin is 1.96, so the interval is 48.04 to 51.96.
2. Why does a small sample need a larger multiplier than 1.96?
The standard deviation is estimated from the same small sample, which adds uncertainty, so the t distribution has heavier tails.
3. A test has p = 0.03. Does that mean the null is 3 per cent likely to be true?
No. It means data this extreme would occur about 3 per cent of the time if the null were true.
4. You run 10 independent tests at the 5 per cent level with no real effects. What is the chance of at least one false positive?
1 − 0.95 to the power 10 = 1 − 0.599 = 0.401, about 40 per cent.
5. Two groups of 40 and 50 differ by 0.8 days with a standard error of 0.46. Is the difference significant at 5 per cent?
t = 0.8 ÷ 0.46 = 1.74, within about 2, so no. The p-value is about 0.09.
Try this
Pick a real comparison you can make with data: two branches, two time periods, two versions of a page, or before and after a change. Set aside forty-five minutes.
1. Write the null and alternative hypotheses in one sentence each, before looking at the numbers.
2. Compute the mean, standard deviation and count for each group. Compute the difference.
3. Compute the standard error of the difference and a 95 per cent confidence interval for it. Check it with the simulators.
4. Compute the t statistic by hand, then check the p-value with scipy.stats.ttestind(a, b, equalvar=False).
5. Compute an effect size, such as the difference in means divided by the pooled standard deviation, and write it in plain words.
6. If your data is categories, build a two-by-two table and compute the chi-square statistic by hand, as in the approvals example, then check with chi2contingency.
7. Count how many comparisons you made while exploring, and work out the Bonferroni level for that many.
8. Write a four-sentence conclusion that says what you found, how big it is, how sure you are, and what you would collect next.
A good result states the interval around the difference, the p-value, and the effect size together, and makes no claim stronger than they support. In the next lesson you meet correlation, the bootstrap, and the A/B test, where these ideas meet randomised experiments.
Tags: Data Science Hypothesis testing Confidence intervals t-test p-value