Lesson 2.2: Sampling, bias and the central limit idea
Learn how a sample tells you about a population, why bias beats sample size, and how the central limit theorem makes averages predictable.
By the end of this lesson you will be able to explain the difference between a population and a sample, choose a sensible way to draw a sample, spot the bias that makes a large sample useless, and use the standard error and the central limit idea to say how far a sample average is likely to be from the truth. You will work out how many survey responses you need for a given margin of error, and see why quadrupling a sample only halves the error. You should have read Lesson 2.1, since this lesson uses the normal distribution and z-scores from it. Every formula here is short and is worked with numbers before it is used.
Almost all of data science is the study of a few examples in order to say something about many. A lender looks at 5,000 past loans to judge what will happen to the next 50,000. A product team measures the behaviour of 10,000 users to decide what to build for ten million. The step from the few to the many is called inference, and it is only safe when you know how the few were chosen and how much they can be trusted. This lesson gives you both.
Population and sample
The population is the whole group you want to know about: all customers, all loans, all transactions next month. A sample is the part of it you actually observe. A number that describes the population, such as its true average income, is a parameter. The same kind of number computed from the sample is a statistic. The parameter is fixed and usually unknown. The statistic changes from one sample to the next, because each sample contains different people.
This difference between the two is the root of everything that follows. If you draw two samples of 100 customers and compute the average spend in each, you will get two different numbers, and neither is exactly the population average. The gap between a statistic and the parameter it estimates is the sampling error. It is not a mistake. It is the unavoidable result of looking at part of the group, and the good news is that for a properly drawn sample it can be measured.
Sometimes the "sample" is not a sample at all. If you have the records of every customer who ever applied, you hold the whole population of applicants. But if you want to say something about future applicants, your data is a sample of a process that continues, and the same cautions apply.
How to draw a sample
How you choose the sample matters far more than how big it is. The gold standard is a random sample, in which every member of the population has a known chance of being chosen. Randomness protects you from your own preferences and from hidden patterns. There are four main designs.
Design · How it works · Good when · Watch out forSimple random · Every member has an equal chance, like drawing names from a hat · A complete list of the population exists · Small groups may be missed by chance
Stratified · Split the population into groups (strata), then sample from each · Groups differ and you need each represented · Needs the group of each member in advance
Cluster · Pick whole groups, such as branches or villages, then survey everyone in them · A list of individuals is costly, travel is costly · Members of a cluster resemble each other, so you learn less per person
Systematic · Take every kth record from an ordered list · A quick, even spread over a file · A pattern in the order that matches k
Against these stand the convenience sample, which is whoever was easy to reach, and the voluntary response sample, where people choose to take part, such as an online poll. These are tempting and cheap, and they are the commonest source of misleading numbers, as the next section shows.
In pandas, a simple random sample and a stratified one take a line each.
python
import pandas as pd
df = pd.readcsv("customers.csv")
simple = df.sample(n=1000, randomstate=42) # simple random sample
stratified: keep 5% of each region, so every region is represented
strat = df.groupby("region", groupkeys=False).sample(frac=0.05, randomstate=42)
The randomstate argument fixes the random choices so that you or a colleague can reproduce the same sample later. Reproducibility is a habit worth forming from the beginning.
Bias: when the sample does not look like the population
Bias is a systematic tilt: the sample is not just noisy, it is wrong in a consistent direction. Taking more of the same kind of data does not remove it. The most common forms are worth knowing by name.
Selection bias: the way people get into the sample is linked to what you are measuring.
Non-response bias: the people who answer differ from those who do not.
Survivorship bias: you only see the cases that lasted, such as companies still operating or loans that were approved.
Measurement bias: the way you measure pushes answers one way, such as leading survey questions or a faulty sensor.
A worked example shows how large the damage can be. A bank wants to know customer satisfaction on a scale of 1 to 5. It shows a survey inside its mobile app. Among its 100,000 customers, 30 per cent use the app and 70 per cent do not. App users rate the bank 4.4 on average, and non-app customers would rate it 3.6 if they were asked. The true satisfaction across all customers is 0.3 × 4.4 + 0.7 × 3.6 = 1.32 + 2.52 = 3.84. The survey, answered only by app users, reports 4.4. That overstates satisfaction by 0.56 points, a gap that no number of additional responses will close. A sample of 50,000 app users is just as wrong as a sample of 500.
Predict first: a company surveys customers who called its helpline and finds that 85 per cent are satisfied. What kind of bias might push this number up or down? Think before reading on.
Customers who call a helpline have already had a problem, so they may be less satisfied than average, and the ones who had a good experience may not respond at all. But those who call may also be the ones who bothered to engage and had their issue solved. The direction is unclear, and that is the point: the sample is not the population, and the survey cannot tell you about customers who never called. The honest statement is "85 per cent of customers who called and answered were satisfied", not "85 per cent of customers are satisfied".
Two further examples from lending show the same pattern. A model built only on approved loans cannot learn how rejected applicants would have behaved, which is a form of survivorship bias known as the reject inference problem. And a sample of this month's defaulters will under-represent defaults that take longer to appear, because some of this month's borrowers have not yet had time to default. In both cases the problem is in the selection, not the sample size.
The defences are simple to state and harder to apply. Draw randomly where you can. Compare the sample with what you know about the population, such as the shares by region or age, and see whether they match. Where groups are over- or under-represented, weight the responses so that each group counts in proportion to its size in the population. In the app example, weighting by the known 30:70 split would need some answers from non-app customers, which is why you need a way to reach them. And always say what the sample covers.
How much can a sample average be off? The standard error
Suppose the sample was drawn properly. How far is its average likely to be from the true population average? The answer is the standard error. For a sample mean it is the population standard deviation divided by the square root of the sample size.
standard error = standard deviation ÷ √n
The standard deviation here describes the spread of individual values, which you know from Lesson 1.3. The standard error describes the spread of the sample average across many possible samples. It is smaller, and it shrinks as n grows, but only with the square root.
Take monthly incomes with a standard deviation of ₹20,000. With 100 people the standard error is 20,000 ÷ √100 = ₹2,000. With 400 people it is 20,000 ÷ √400 = ₹1,000. Four times as many people halves the error. To cut the error to a tenth you need a hundred times as many people. This square root law is why surveys of 1,000 people are so common: going from 1,000 to 10,000 costs ten times as much and only improves the error by about a factor of three.
A convenient rule is that the sample average is within about two standard errors of the truth in about 95 per cent of samples. The precise multiplier is 1.96, from the normal distribution. This range, the margin of error, is the basis of the confidence intervals in Lesson 2.3.
The chart makes the lesson visible. The first fourfold step, from 25 to 100 people, removes 2 thousand rupees of error. The step from 1,600 to 6,400 removes only a quarter of a thousand, at a hundred times the cost of the first. Beyond a certain size, effort is better spent on reducing bias than on adding rows.
The law of large numbers
The standard error formula contains a promise. As n grows, the standard error shrinks towards zero, so the sample average gets closer to the population average. This is the law of large numbers. If you flip a fair coin 10 times, the share of heads can easily be 30 or 70 per cent. After 10,000 flips it will be very close to 50 per cent. The typical size of the error in the share of heads is the square root of 0.25 ÷ n, which is a handy way to see the law at work.
The law of large numbers is what makes insurance and lending work as businesses. Any single policy or loan is unpredictable. A book of a hundred thousand is predictable, because the average outcome settles down. But the law has a limit: it needs the outcomes to be independent and the sample to be unbiased. A recession makes loans fail together, and no amount of diversification across loans in the same economy removes that.
The central limit theorem
Here is a remarkable fact. Take any population, however lopsided, and draw many random samples of the same size n. Compute the average of each sample. Those averages will form a distribution that gets closer to a bell curve as n grows, centred on the population mean, with a spread equal to the standard error. This is the central limit theorem, and it holds whatever shape the original data had, as long as it has a finite spread.
An everyday example is a die. A single roll is flat: each of 1 to 6 is equally likely, with a mean of 3.5 and a standard deviation of 1.71. Now roll the die 30 times and take the average. Over many such sets of 30 rolls, the averages pile up around 3.5 in a bell shape, with a standard error of 1.71 ÷ √30 = 0.31. The averages are much more predictable than the single rolls, and much more bell-shaped than the flat distribution of one roll.
This is why the normal distribution appears so often in practice and why the bell curve is useful even for data that is not normal. You can ask what the chance is that the average of 30 rolls exceeds 4. The z-score is (4 − 3.5) ÷ 0.31 = 1.6, which leaves about 5.5 per cent above it. Try other numbers.
You can see the theorem at work in a short simulation. It draws a very skewed population, repeats the sampling many times, and shows that the averages are roughly bell-shaped even though the population is not.
python
import numpy as np
rng = np.random.defaultrng(0)
population = rng.exponential(scale=10, size=1000000) # skewed: long right tail
means = [rng.choice(population, size=50).mean() for in range(5000)]
print(population.mean(), population.std()) # about 10 and 10
print(np.mean(means), np.std(means)) # about 10 and 10/sqrt(50) = 1.41
If you plot a histogram of means you will see a bell curve, although the population is nothing like one.
When the central limit idea fails
The theorem is a statement about large samples, and "large" depends on the data. For roughly symmetric data, a sample of 30 is often enough. For heavily skewed data, such as claim amounts or the sizes of transactions, you may need hundreds or thousands before the average behaves. It also fails when values are dependent, as in a time series where each day resembles the day before, when the data has extremely heavy tails so that a single value can dominate, and when the sample is tiny. Always look at the data before trusting a bell-curve shortcut.
How big a sample do you need?
Before a survey or test you must decide how many people to ask. The answer depends on the margin of error you can tolerate. For a proportion, such as the share who prefer a feature, the number needed is
n = 1.96² × p × (1 − p) ÷ margin²
where p is the expected share, written as a fraction, and the margin is also a fraction. If you do not know p, use 0.5, which gives the largest and so the safest n. For a margin of ±5 percentage points, n = 3.84 × 0.25 ÷ 0.0025 = 384.2, so you need 385 people. For ±3 points you need 1,068. For ±2 points you need 2,401. Notice how quickly the number rises: halving the margin from 4 to 2 points multiplies the sample by four.
Two practical points. First, this formula assumes a random sample, so a large non-random sample can be less trustworthy than a small random one. Second, when the sample is a big share of a small population, such as 500 people out of a population of 2,000, the effective error is smaller than the formula says, and a correction for the finite population applies. For populations of thousands or millions it makes no difference.
Pulling the ideas together
It helps to see the pieces as a chain. You want a number about a population. You draw a sample, ideally at random, so that bias is controlled. You compute a statistic, such as the mean. The statistic varies from sample to sample, and its variation is the standard error, which shrinks with the square root of n. The central limit idea tells you that, for large enough samples, the average is roughly normal around the truth, so you can say how far off you are likely to be. The next lesson turns that sentence into confidence intervals and tests.
Idea · What it tells you · Fixed by a bigger sample?Sampling error · Random difference between sample and population · Yes, slowly, with the square root of n
Bias · Systematic tilt from how the sample was chosen · No
Standard error · Typical size of the sampling error · It shrinks as n grows
Margin of error · About 2 standard errors, the 95% range · It shrinks as n grows
## Common mistakes
Trusting a big sample because it is big. A biased sample of a million is worse than a random sample of a thousand. Fix the selection first.
Treating an online poll as a sample. People who choose to answer differ from those who do not.
Confusing the standard deviation and the standard error. The first describes individuals and does not shrink with n. The second describes the average and does.
Expecting linear gains from more data. To halve the error you need four times as many rows.
Using the central limit idea with tiny or dependent samples. Check the shape, the size and the independence of the data.
Forgetting non-response. If only 20 per cent answer, plan for it and look at who they are.
Not saying what the sample covers. State the population the result applies to, and no wider.
Key takeaways
A population has parameters, a sample has statistics, and the gap between them is sampling error, which can be measured when the sample is random.
Bias is a systematic error from how a sample is chosen, and it is not reduced by collecting more of the same.
The standard error of a mean is the standard deviation divided by the square root of n, so quadrupling the sample halves the error.
The law of large numbers makes averages settle as samples grow. The central limit theorem says averages become roughly bell-shaped, whatever the shape of the data.
For a proportion, about 385 responses give a margin of ±5 points, 1,068 give ±3 and 2,401 give ±2, using the worst case p of 0.5.
Compare a sample to what you know about the population, weight where needed, and say clearly who the result applies to.
Check yourself
1. A survey of app users reports an average satisfaction of 4.4, but the bank's customers are 30 per cent app users with 4.4 and 70 per cent others with 3.6. What is the true average?
0.3 × 4.4 + 0.7 × 3.6 = 3.84. The app-only survey overstates it by 0.56 because of selection bias.
2. Individual incomes have a standard deviation of ₹30,000. What is the standard error of the average for 900 people?
30,000 ÷ √900 = 30,000 ÷ 30 = ₹1,000.
3. If you want to halve the margin of error of a survey, how many times more responses do you need?
Four times as many, because the error falls with the square root of n.
4. A sample of 1,000 people has p = 0.5 for a yes-or-no question. What is the approximate margin of error in percentage points?
1.96 × √(0.25 ÷ 1,000) = 1.96 × 0.0158 = 0.031, about ±3.1 points.
5. A bank checks only approved loans to see how well its model performs. What kind of bias is this, and why does it matter?
Survivorship, or selection, bias. Rejected applicants are not in the data, so you cannot tell how they would have behaved or how well the model sorts them.
Try this
Use any dataset with at least a few thousand rows, such as an open data file, or build one with the simulation above. Set aside forty to forty-five minutes.
1. Treat the whole table as the population and compute the mean of one numeric column. Write it down as the true value.
2. Draw a simple random sample of 30 rows and compute its mean. Repeat 20 times. Look at how much the 20 means differ from each other and from the true value.
3. Repeat step 2 with samples of 120 rows. Compare the spread of the means. If the sample size is four times larger, the spread should be about half.
4. Compute the standard deviation of the column and the standard error for n = 30 and n = 120, and compare them with the spread of the sample means you observed.
5. Make a deliberately biased sample, for example only the rows above the median of some other column, and compute its mean. How far is it from the true value, and does a larger biased sample help?
6. Use the survey simulator to find the number of responses you would need for a margin of ±4 points on a yes-or-no question, and the cost at a rate of your choice.
7. Write five sentences summarising what you found about sample size, bias and the square root rule.
A good result is a small table of sample sizes against the spread of the means, with one sentence stating the square root pattern, and one stating why bias survived a bigger sample. In the next lesson these ideas become the confidence interval and the hypothesis test, the two tools analysts use to say how sure they are.
Tags: Data Science Sampling Central limit theorem Bias Statistics