Selection bias
Selection bias is the distortion that appears when the cases you measured differ systematically from the cases you meant to describe. It is a problem of who got into the sample, not of how the sample was analyzed, and no amount of careful arithmetic afterward can undo it. A biased sample of one million produces a precise answer to a question nobody asked.
The distortion is invisible from inside the data. Everything about a biased sample looks normal: the responses are real, the measurements are accurate, the margin of error is small. What is missing does not appear as a gap or an error message. It appears as a plausible number, and the only way to detect it is to reason about the process that decided who was included.
What selection bias is
Selection bias is any systematic difference between the sampled group and the target population that arises from the selection process itself. The target population is whoever the conclusion is about. The sampled group is whoever the method could actually reach and whoever, having been reached, actually took part. When those two groups differ on anything related to what is being measured, every estimate from the sample is shifted.
What distinguishes selection bias from random sampling error is that it does not shrink. Random error falls with the square root of the sample size, so a sample of 10,000 has a margin of error roughly one tenth that of a sample of 100. Selection bias is a constant offset: it stays the same size at 100 respondents and at 100,000, and increasing the sample simply narrows a tight interval around the wrong value. That is why sample size and selection bias are separate questions, and why the first is a poor answer to the second.
The one-sentence version: ask who could not have been counted
In one sentence: the useful question is never who is in the sample, it is who could not have got in. The people the method structurally excludes are the people who determine the size of the bias.
The clearest illustration in the record is the work of the statistician Abraham Wald for the Statistical Research Group at Columbia University during the Second World War. Asked how to distribute armor on aircraft, analysts had the damage patterns of planes that returned from missions, which concentrated in certain areas of the airframe. Wald's memoranda set out the point that the data were censored by the selection process: the sample contained only aircraft that survived their damage. Areas showing damage on returning planes were, by that fact, areas a plane could be hit in and still come home. The reasoning is about which planes could enter the sample, not about the damage that was measured, and it is worth knowing that this came from a series of technical memoranda rather than from the illustrated version that circulates online.
The arithmetic worked through: a survey with a 4 percent response rate
Work the arithmetic on a satisfaction survey emailed to 20,000 customers, of whom 800 respond, a response rate of 4 percent. Among the 800 respondents, 480 report a serious problem with the product, which is 60 percent, and the headline writes itself. These are invented illustrative figures.
Now reason about the 19,200 people who did not respond. Nothing forces them to resemble the respondents, and there is a specific reason to expect they do not: annoyance is a strong motive to open a survey and indifference is not. Suppose the true problem rate among non-respondents is 10 percent.
- Problems among respondents: 60 percent of 800, which is 480.
- Problems among non-respondents: 10 percent of 19,200, which is 1,920.
- Problems in the full customer base: 480 plus 1,920, which is 2,400.
- True rate across all 20,000 customers: 2,400 divided by 20,000, which is 12 percent.
The survey reported 60 percent and the population figure is 12 percent, a factor of five, with no measurement error anywhere in the process. Note also that the reported figure carries a margin of error of about 3.5 points on 800 responses, so it would be published as 60 percent plus or minus 3.5. The precision is genuine and it is precision about the wrong group.
Loss to follow-up produces the same arithmetic inside a study rather than at recruitment. Begin a twelve-month program with 1,000 participants and finish with 600, having lost 400. If 180 of the 600 completers improved, the completer rate is 30 percent. If the 400 who left did so because the program was not working for them, and none of them improved, the rate across everyone who started is 180 divided by 1,000, which is 18 percent. The completers were not a random subset. They were selected by the outcome being measured, which is the definition of the problem.
What a failure looks like: five ways a sample goes wrong
Selection bias failures fall into five recognizable types, and naming the type usually tells you which direction the estimate is shifted.
| Type | How the sample gets distorted | Who is missing |
|---|---|---|
| Convenience sampling | Whoever was easiest to reach is measured | Anyone outside the channel used, such as a mailing list or a single site |
| Self-selection | Participants opt in, and motivation correlates with the outcome | The indifferent majority |
| Nonresponse | A valid frame is used but only some respond | Everyone whose reason for silence relates to the question |
| Loss to follow-up | Participants leave before the final measurement | Those for whom things went worst, or best, or who moved |
| Survivorship | Only cases that lasted long enough to be observed are available | Failures that ended before the observation window |
Survivorship is the type that produces the most confident wrong advice, because the surviving cases are exactly the ones that get studied. A study of companies that have lasted twenty years, or of products still on sale, or of methods still in use, is a study of a filtered population, and the filter did the selecting on the outcome of interest. Whatever those survivors have in common may be the cause of survival, or it may be a feature that the failures also had and took with them.
The common misreading: sampling bias, selection bias and sampling error
The three terms are used loosely and the distinctions are worth holding. Sampling bias is one form of selection bias, specifically the form introduced when the sample is drawn: a frame that omits part of the population, or a recruitment method that reaches one kind of person. Selection bias is the broader term and also covers distortions introduced after recruitment, such as loss to follow-up and selective exclusion during analysis. In epidemiology the broader usage is standard; in survey research the two terms are often treated as synonyms.
Selection bias is not a sampling error, and this is the misreading that does real damage. Sampling error is random, it is quantified by the margin of error, and it averages out across repeated samples. Selection bias is systematic, it is not quantified anywhere in the standard output, and it does not average out: repeat the biased procedure a thousand times and you get a thousand samples wrong in the same direction. A reported margin of error is a statement about sampling error only, and it is silent about selection bias by construction.
A third confusion is with confirmation bias, which is about which evidence a person attends to after it exists. Selection bias is about which evidence exists to be attended to. The two compound: a sample chosen badly, then read selectively, produces a conclusion that is wrong twice over.
The test to run on any sample
Run these five questions on any study, survey or dataset before you accept what it reports.
- Name the population the conclusion is about, in one sentence.
- Describe the route by which someone entered the sample, step by step.
- Identify who that route excludes entirely, and whether their exclusion relates to the outcome.
- Find the response rate and the dropout rate, and treat any figure below about 30 percent as a warning.
- Ask what the result would look like if the missing group were the opposite of the measured one.
Limiting selection bias, and the methods that try to correct it
Selection bias is limited at the design stage far more effectively than it is corrected afterward, and four measures do most of the work.
- Define the sampling frame before recruitment, and record what it excludes.
- Randomize selection within the frame rather than accepting whoever volunteers.
- Report the response rate and the dropout rate alongside every headline figure.
- Analyze by original assignment, counting everyone who started rather than everyone who finished.
Correction after the fact is possible but it always requires an assumption that cannot be checked from the data. Weighting adjusts the sample so its known characteristics match the population, which works only for characteristics you measured and only if they capture the reason for nonresponse. The economist James Heckman developed a formal correction for sample selection in the late 1970s, modeling the selection process itself as a second equation, work recognized by the Nobel prize in economics in 2000. In machine learning, a line of work published in the mid 2000s addresses the same problem by reweighting training examples so that their distribution matches unlabeled data drawn from the target population, an approach known as kernel mean matching.
Each of these methods trades an unverifiable assumption for an estimate, and that is the honest limit. Correction reduces bias when the model of the selection process is roughly right and increases it when the model is wrong, with no signal in the output to tell you which happened. Reading numbers covers the wider habit of asking what a figure was measured on, and The null hypothesis supplies the question a corrected estimate is still obliged to answer. The gambler's fallacy is a neighboring error in the same family: reasoning about a specific case while ignoring how the case came to your attention in the first place.
Why there is no selection bias calculator, and what to compute instead
There is no selection bias calculator and there cannot be one, because the quantity it would have to measure lives entirely outside the data. Every calculator works on numbers you have. Selection bias is a statement about the people you do not have, and no arrangement of the collected responses contains information about the ones that were never collected. Tools that claim to compute it are computing something else, usually the margin of error, which is a measure of random noise and is silent about systematic exclusion by construction.
What you can compute is a sensitivity analysis, which asks what the missing group would have to be like for the conclusion to survive. Run it on the survey above, where 800 of 20,000 customers responded and 480 of them, or 60 percent, reported a problem.
- For the true rate across all 20,000 customers to be 30 percent, the total would need to be 6,000 problems. Subtracting the 480 known cases leaves 5,520 among 19,200 non-respondents, a rate of 28.75 percent.
- For the true rate to be 15 percent, the total would be 3,000, leaving 2,520 among the non-respondents, a rate of 13.1 percent.
- For the headline figure of 60 percent to be right about the customer base, the 19,200 silent customers would have to report problems at 60 percent as well, which is the assumption the headline is quietly making.
Two decisions have to be made before any of this is meaningful, and neither is arithmetic. You have to name the population the claim is about, and you have to describe the route by which a person entered the sample. Those two sentences do more work than any computation on this page, and they are the reason selection bias is caught by reasoning about a process rather than by running a procedure on a spreadsheet.
What a sensitivity analysis does not settle is what the missing group is actually like. It converts an argument about bias into a statement of the form "this conclusion holds only if the silent 96 percent resemble the vocal 4 percent within about this much", and then hands the judgment back to you. Weighting and formal selection models go a step further and produce a corrected number, which is a real improvement and also a trap, because the corrected number looks exactly as authoritative as an uncorrected one and rests on an assumption about the selection process that the data cannot test. A tool returns a figure. Whether the figure describes the population you care about is a question about recruitment, and it is answered before the first response arrives.