The base rate fallacy
The base rate fallacy is the error of judging how likely something is from how well the evidence fits, while ignoring how common the thing was to begin with. It is the reason a test that is right 99 percent of the time can produce positive results that are wrong half the time, and the reason that arithmetic surprises almost everybody the first time they see it worked through.
The base rate is simply the underlying frequency: how many items in the population actually have the property before anybody tests anything. When the base rate is low, the small error rate of a good test is applied to a very large group of negatives, and the resulting false positives can easily outnumber the true ones. Nothing about the test is broken. The fallacy is in the reading.
What the base rate fallacy is
The base rate fallacy is a reasoning error in which a person updates on the strength of new evidence without weighting it by the prior frequency of what the evidence points to. It sits in the family of judgment errors described by the psychologists Daniel Kahneman and Amos Tversky in the 1970s, alongside the availability heuristic and the conjunction fallacy, and it is the one with the sharpest arithmetic behind it.
What distinguishes the base rate fallacy from its siblings is that it has an exact answer. Availability is about what comes to mind easily and the conjunction fallacy is about a compound event being judged more likely than one of its parts. Both are demonstrated by comparison. The base rate fallacy is demonstrated by a single calculation with one right result: at a base rate of 1 percent and a test accuracy of 99 percent, exactly half of all positive results are false. That number is not an estimate. It falls out of a table of 10,000 items.
The one-sentence version, and how to recognize it in the wild
In one sentence: how well the evidence fits tells you nothing until you know how rare the thing is. Carry that sentence and you have the fallacy. Everything else on this page is the demonstration.
You are looking at a probable base rate fallacy whenever an argument moves from an accuracy figure straight to a conclusion about an individual case. The tell is a sentence of the form "the detector is 99 percent accurate, so this flagged item is almost certainly defective". The first clause is about the test. The second is about a specific item. Getting from one to the other requires the base rate, and if nobody has mentioned it, nobody has done the calculation. Three phrasings that should stop you: "the system is right 99 times out of 100", "false positives are only 1 percent", and "the match probability is one in a million".
The arithmetic worked through: 10,000 circuit boards
Work the arithmetic on 10,000 circuit boards, of which 1 percent are genuinely defective, tested by an inspection system that is 99 percent accurate in both directions. These are invented illustrative figures, chosen because they divide cleanly. Accuracy in both directions means the system flags 99 percent of the boards that really are defective, and clears 99 percent of the boards that really are sound.
Start by splitting the population, then apply the test to each half separately.
- Defective boards: 1 percent of 10,000, so 100 boards.
- Sound boards: the remaining 9,900 boards.
- Defective boards flagged: 99 percent of 100, so 99 true positives. One defect is missed.
- Sound boards flagged: 1 percent of 9,900, so 99 false positives. The other 9,801 are correctly cleared.
| Out of 10,000 boards | Flagged by the system | Cleared by the system | Row total |
|---|---|---|---|
| Actually defective | 99 | 1 | 100 |
| Actually sound | 99 | 9,801 | 9,900 |
| Column total | 198 | 9,802 | 10,000 |
The system flags 198 boards in total, and only 99 of them are defective. So the probability that a flagged board is genuinely defective is 99 divided by 198, which is 50 percent. A coin flip. The reason is visible in the table: the 1 percent error rate is applied to 9,900 sound boards and produces 99 mistakes, which is almost exactly the number of real defects that exist in the whole batch.
Now lower the base rate to 0.1 percent and run 100,000 boards through the same system. There are 100 defective boards and 99,900 sound ones. The system flags 99 of the defects and 1 percent of the sound boards, which is 999 false positives. Of 1,098 flagged boards, 99 are real: about 9 percent. The test did not change. Only the base rate did, and the same result now means something ten times weaker.
The mirror side is worth noting, because it is where the test earns its keep. Of the 9,802 boards cleared in the first table, 9,801 really are sound, so a clear result is right 99.99 percent of the time. A test can be nearly worthless on one answer and near certain on the other.
What a failure looks like in practice
A base rate failure in practice looks like an expensive, confident response to a screening result that was never strong enough to justify it. Take the same inspection system running on a line that produces 10,000 boards a day. Every day it flags 198 boards. If the response to a flag is scrapping the board, 99 sound boards a day are destroyed. If the response is a second, independent inspection, the arithmetic changes completely: applied to the 198 already flagged, where the base rate is now 50 percent rather than 1 percent, a second 99 percent accurate test gives 98 true positives against 1 false positive, and a doubly flagged board really is defective about 99 percent of the time.
That is the practical lesson buried in the fallacy, and it is not "never trust screening". Screening tests are built to be run on populations with low base rates, and a 50 percent hit rate on a 1 percent base rate is a hundredfold improvement in the odds. The failure is not in using the test. The failure is treating the first result as a verdict rather than as a reason to raise the base rate and test again.
The common misreading: accuracy is not the answer
The common misreading is to treat "99 percent accurate" as if it answered the question "given a positive result, how likely is the item to be defective". It answers the reverse question: "given a defective item, how likely is a positive result". Those two conditional probabilities are different numbers, and the gap between them is the whole fallacy. In the first table they are 50 percent and 99 percent.
A second misreading is more subtle. People often assume that ignoring base rates is a failure of arithmetic ability, but the classic experiments suggest it is a failure of attention. Kahneman and Tversky gave participants a short personality description and asked whether the person was more likely to be an engineer or a lawyer, having first told them the proportion of engineers in the sample. Participants who were told the sample was 30 percent engineers gave nearly the same answers as those told it was 70 percent. The stated base rate was available, plainly written, and almost entirely unused. A description that sounds representative crowds it out, which is why the fallacy is also called base rate neglect and is treated as the counterpart to the representativeness heuristic.
The test to run on any accuracy claim
Run this test on the next accuracy claim you meet, and it takes under a minute with a phone calculator.
- Ask: out of every 10,000 cases, how many actually have the property being tested for?
- Ask: of those, how many does the test catch?
- Ask: of the remainder, how many does the test flag by mistake?
- Divide the true positives by all positives, and read the answer as a percentage.
- Ask what would change if the base rate were ten times lower.
If nobody can supply the answer to the first question, the accuracy figure cannot support any conclusion about an individual case, and saying so is the whole of the skeptical move. The wider habit of turning a claim into counts before arguing about it is the subject of Reading numbers, and the formal machinery for asking whether a result would have appeared anyway is the null hypothesis. The gambler's fallacy is the neighboring error in the same family: there, too, people reason about a specific case while ignoring the underlying rate that governs it. Critical thinking exercises put both under practice.
Base rate neglect, representativeness and availability: the MCAT distinctions
In MCAT-style psychology and sociology questions, the base rate fallacy is almost always tested against three near neighbors, and the distinctions are worth holding separately because the exam items turn on them.
- Base rate fallacy and base rate neglect name the same error, with neglect the more common term in psychology writing and fallacy the more common term in logic and statistics.
- Representativeness is the heuristic that causes it: a case that resembles the stereotype of a category is judged likely to belong to that category, regardless of how large the category is.
- Availability is a different heuristic: likelihood is judged by how easily examples come to mind, so vivid or recent events feel more probable than they are.
- The conjunction fallacy is a third relative, in which a detailed, specific scenario is rated more probable than the broader scenario that contains it.
The practical difference is the direction of the mistake. Under availability the frequency estimate itself is wrong. Under the base rate fallacy the frequency may be stated correctly, in the question, in writing, and be discarded anyway in favor of the evidence that fits. That is the harder error to fix, because knowing about it does not stop it. What stops it is writing the counts out of 10,000, every time.
Is there a base rate calculator, and which input decides the answer?
Yes. A base rate calculator exists under several names, including Bayes theorem calculators and positive predictive value tools, and all of them compute the same thing: you enter a prevalence, a sensitivity and a specificity, and it returns the probability that a flagged case is genuinely positive. It is the table from earlier on this page, built for you. The arithmetic is trivial and the tool is honest, which is why the interesting part is the input form.
Two of the three inputs come from the test and one does not. Sensitivity and specificity are properties of the inspection system, measured during its validation and quoted by whoever built it. Prevalence is a property of your population, it is not measured by the test at all, and it is the input the answer swings on. Hold the 99 percent accurate system fixed and vary only the prevalence across 10,000 items:
- At 10 percent prevalence: 1,000 defective, of which 990 are flagged, against 90 false positives from the 9,000 sound items. A flag means defective 91.7 percent of the time.
- At 1 percent prevalence: 99 true positives against 99 false positives, so a flag means defective 50 percent of the time.
- At 0.1 percent prevalence: 99 true positives against 999 false positives, so a flag means defective about 9 percent of the time.
Same test, same accuracy figure, three answers spanning an order of magnitude, decided entirely by the number the calculator cannot supply. Enter a prevalence guessed from impression and the tool will return a confident percentage built on the guess, formatted to one decimal place.
What the output does not settle is whether the prevalence you entered applies to the case in front of you. A population base rate is the right input for an item drawn at random from the line, and the wrong input for an item that reached you because something already looked wrong with it. That item belongs to a pre-screened subgroup with a much higher base rate, and using the general figure understates the risk exactly as badly as ignoring the base rate overstates it. Nor does the output tell you what to do about a flag, since scrapping, re-testing and ignoring have different costs that no probability contains. A calculator converts three numbers into a fourth. Deciding which population the case came from is the judgment it hands back to you.