Statistical significance
Statistical significance means that a result would be unlikely to appear by chance alone if there were no real effect, and nothing more than that. It is a statement about how surprising the data would be under a specific assumption of no effect. It is not a measure of how large the effect is, how important it is, or how probable it is that the claim being tested is true.
Almost every popular misuse of the term comes from one substitution: reading a p value of 0.03 as "there is a 3 percent chance the result is a fluke" or "a 97 percent chance the effect is real". Neither reading is correct, and the correct reading is only slightly harder to hold. The p value is computed by assuming there is no effect and asking how often data this extreme would turn up anyway.
What statistical significance is
Statistical significance is a decision rule applied to a probability, not a property that a finding possesses. The probability is the p value: the chance of observing a result at least as extreme as the one obtained, calculated on the assumption that the null hypothesis is true. The decision rule is the significance level, usually written as alpha and usually set at 0.05. When the p value falls below alpha, the result is called statistically significant and the null hypothesis is rejected.
What distinguishes significance from every other quantity in a results table is that it depends on sample size as much as on the effect. A difference of 0.1 percentage points can be highly significant in a sample of 500,000 and invisible in a sample of 2,000, even though the difference itself is identical. That single fact explains most of the confusion that follows, and it is why a significant result and an important result are different questions.
Is the significance level the same as the p value?
No. The significance level is a threshold you choose before collecting data; the p value is a number the data produce. Alpha is a policy, typically 0.05, occasionally 0.01 for work where a false alarm is costly. The p value is a measurement. Comparing the two gives the verdict, in the same way that comparing a measured length to a tolerance gives a pass or a fail.
The confusion is understandable, because both are decimals between 0 and 1 and both get called "the significance". Keeping them apart matters in practice for one reason: alpha must be fixed in advance. A researcher who collects data, sees a p value of 0.07, and decides that 0.1 is a reasonable threshold after all has not run a test at the 0.1 level. They have run a test with no threshold at all, and confirmation bias does the rest.
The arithmetic worked through: ten flips of a suspect coin
Work the arithmetic on ten flips of a coin that lands heads nine times. The question is whether that result is evidence the coin is biased. The null hypothesis is that the coin is fair, so each flip has probability 0.5 of heads, and the p value is the probability of getting a result at least as lopsided as nine heads if that null is true.
Ten flips have 2 to the power of 10 possible sequences, which is 1,024, and each is equally likely under the null. Count the ones that are at least as extreme:
- Sequences with exactly 10 heads: 1.
- Sequences with exactly 9 heads: 10, one for each position the single tail can occupy.
- Total at least as extreme in the heads direction: 11 of 1,024, which is 0.0107.
- Counting equally extreme results in the tails direction as well: 22 of 1,024, which is 0.0215.
So the two-sided p value is 0.0215, or about 2.2 percent. Against the usual alpha of 0.05 the result is statistically significant, and the null hypothesis of a fair coin is rejected. Read what that sentence does and does not say. It says: if the coin were fair, data this lopsided would turn up about 2 times in 100 runs of ten flips. It does not say there is a 2.2 percent chance the coin is fair. That second statement requires knowing how many coins in the world are biased before you started, which is a base rate, and no p value contains one.
Now the same arithmetic with the threshold in view. Run 20 independent tests on data with no real effect in it at all, each at alpha 0.05. Each test has a 95 percent chance of coming out non-significant, so the chance that all 20 do is 0.95 multiplied by itself 20 times, which is 0.359. The chance of at least one significant result from 20 tests of nothing is about 64 percent. Significance at 0.05 is not a rare event. It is a one in twenty event, repeated as often as somebody chooses to repeat it.
What a failure looks like: significant and trivial at the same time
A failure of significance reasoning most often looks like a real, correctly computed, statistically significant result that means almost nothing. Consider a website test with 500,000 visitors seeing each of two page designs. Version A converts at 4.00 percent, which is 20,000 conversions. Version B converts at 4.12 percent, which is 20,600 conversions. The gap is 600 conversions, and on samples that large it clears the 0.05 threshold comfortably.
Now ask whether 0.12 percentage points is worth anything. That depends entirely on what a conversion is worth and what the redesign cost, and the p value has no opinion on either. Shrink the same test to 2,000 visitors per version and the expected counts are 80 and 82.4 conversions, a gap of two or three cases that no test would distinguish from noise. Identical effect, opposite verdict, because significance answers "could this be chance" and not "is this worth doing".
The failure also runs the other way. A non-significant result on a small sample is routinely reported as showing no effect, when it shows only that the study was too small to detect one. That is a question about sample size and statistical power, and it is the reason a null result from an underpowered test is nearly uninformative.
The common misreading: what a p value is not
The common misreading treats the p value as the probability that the hypothesis is false. It is not, and it cannot be, because it was calculated by assuming the null hypothesis is true. A quantity computed under an assumption cannot also be the probability that the assumption holds. Four specific misreadings are worth being able to name.
- The p value is not the probability that the null hypothesis is true.
- The p value is not the probability that the finding will replicate.
- A p value of 0.049 and a p value of 0.051 do not describe meaningfully different worlds, despite falling on opposite sides of the threshold.
- Failing to reach significance is not evidence that the effect is zero, only that this test did not distinguish it from zero.
There is one further misreading that is really a research practice. Because a threshold of 0.05 can be reached by testing enough variations, the significance of a result depends on how many analyses were run before this one was reported. A test chosen after seeing the data is not the same test as one specified before, and the reported p value no longer means what its formula says it means.
The test to run on any significant finding
Run these five questions on the next result described as statistically significant, and notice that only the first one is about the p value at all.
- Ask what the effect size is, in the units of the thing being measured, not in p values.
- Ask how many outcomes and subgroups were tested before this one was reported.
- Ask what the sample size was, and whether the threshold was set before the data arrived.
- Ask what the null hypothesis actually was, since a rejected null is only informative if it was a sensible one.
- Ask whether a difference this size would change any decision you would make.
The fifth question is the one that separates statistical significance from practical significance, and it is the only one whose answer is yours rather than the analyst's. A result can be significant and useless, and it can be non-significant and worth pursuing further with a larger study. Turning the finding back into raw counts before you argue about it is the habit taught in Reading numbers, and the assumption being rejected is set out in The null hypothesis. The gambler's fallacy shows the same threshold error in a different costume: a run that looks too extreme to be chance, in a setting where chance produces such runs constantly.
Is there a statistical significance calculator, and what does it decide for you?
Yes. A statistical significance calculator exists in many forms, including online p value tools, A/B testing calculators, chi square calculators and the built-in functions of any spreadsheet, and what every one of them computes is narrow and mechanical. You supply counts or means, a sample size and a test; it returns a p value and compares it to a threshold you gave it. That is the entire service. The arithmetic is the easy part of significance testing and it is the only part that has been automated.
Four decisions have to be made before a calculator can answer anything, and it will accept whatever you enter without comment: which test fits the data, whether the comparison is one-sided or two-sided, what the significance level is, and how many comparisons are being run in total. The one-sided toggle alone moves the answer by a factor of two. The nine-heads-in-ten result above returns 0.0107 one-sided and 0.0215 two-sided from identical data, because the two-sided version also counts the equally lopsided outcomes in the tails direction. A calculator offering that switch is asking a question it cannot answer for you, and choosing the side after seeing which one clears 0.05 is not a test of anything.
What the output does not settle is everything that made the question worth asking. It does not report the effect size, so a p value of 0.001 is compatible with a difference too small to act on. It does not know how many analyses preceded this one, so it cannot adjust for the fact that a threshold of 0.05 is reached by one test in twenty on noise alone. It does not know whether the null hypothesis you tested was a sensible one to test. And it cannot tell you whether a difference of this size would change any decision you would make, because that is a judgment about the world rather than a calculation on the data. A calculator returns a number. Whether the number means what you want it to mean is not a computation, and no tool has ever performed it.