Research Literacy
What Is an Effect Size, and Why Does It Matter More Than the P-Value?
The short answer
A p-value tells you how likely a result would be if there were truly no effect at all — it says nothing about how large that effect actually is. In a 1989 trial of 22,071 physicians, low-dose aspirin cut heart attack risk by 44% (p<0.00001): a result that was both statistically significant and clinically large. In a 2020 Cochrane review of 86 omega-3 trials and 162,796 participants, the reduction in coronary heart disease death was also technically significant (95% CI just barely excluding no effect) — but it worked out to a number needed to treat of 334, meaning 334 people would need to take the supplement to prevent a single death. Two 'significant' results, two very different sizes of real-world benefit.
Quick answers
- What is it?
- An effect size (a relative risk, mean difference, or number needed to treat) measures how large a treatment's real-world impact actually is, separately from whether a p-value crossed the significance threshold.
- Why are people interested in it?
- Two results can both be called "statistically significant" while one changes almost nothing for a patient and the other changes a great deal, and readers need to know how to tell the difference.
- What has actually been studied?
- The 1989 Physicians' Health Study (22,071 physicians) testing aspirin, showing three different outcomes with three different combinations of effect size and significance, and the 2020 Cochrane omega-3 review (86 trials, 162,796 participants) with its number-needed-to-treat of 334.
- How strong is the evidence?
- Grounded in two large, well-documented studies used as direct illustrations of the concept: a 44% relative reduction in heart attacks paired with a tiny p-value, versus a technically "significant" coronary death benefit that required treating 334 people to help one.
- What don't we know yet?
- Why the aspirin trial's stroke signal didn't reach significance despite a nearly doubled point estimate, whether a number needed to treat of 334 conceals more meaningful benefit in a narrower subgroup, and how much of the omega-3 coronary finding would survive another decade of new trials.
- Where can I read the original research?
- Sources at the bottom of this page: Physicians' Health Study Steering Committee, New England Journal of Medicine, 1989; and Abdelhamid AS, Brown TJ, Brainard JS, et al., Cochrane Database of Systematic Reviews, 2020.
The short answer: A p-value answers a narrow, specific question: if this treatment truly did nothing, how likely would a result this extreme be, just from chance? It says nothing about how big the effect is, or whether it's large enough to matter to an actual person. An effect size — a relative risk, a mean difference, a number needed to treat — answers that second question directly. The two don't move in lockstep. A landmark aspirin trial found both a large effect and a vanishingly small p-value for one outcome, and no effect at all for another, in the very same patients. A massive pooled review of omega-3 supplements found a "statistically significant" benefit that required treating 334 people to help one.
What the research shows
What a p-value actually measures
A p-value is the probability of observing a result at least as extreme as the one a study found, assuming the treatment being tested has zero true effect. Convention treats p<0.05 as the line for "statistically significant" — but that threshold says nothing about magnitude. A p-value can be tiny because an effect is genuinely large, or because a sample is enormous enough to detect even a trivial, practically meaningless difference with confidence. Effect size measures the second, separate question: how much did the outcome actually change? A relative risk of 0.56 and a relative risk of 0.97 can both be "statistically significant" in a large enough trial, and they describe wildly different realities for a patient deciding whether to act on the finding.
A trial where both lined up — and one where they didn't
The Physicians' Health Study, published in the New England Journal of Medicine in 1989, randomized 22,071 physicians to low-dose aspirin or placebo and followed them for an average of 60.2 months. For heart attack specifically, the result was unambiguous on both counts: relative risk 0.56 (95% CI 0.45–0.70), a 44% reduction, with a p-value below 0.00001 — annual incidence of 254.8 per 100,000 in the aspirin group versus 439.7 in the placebo group. Large effect, tiny p-value, in agreement.
But the same trial, in the same patients, found something different for overall cardiovascular mortality: relative risk 0.96 (95% CI 0.60–1.54) — a confidence interval so wide it comfortably includes both a real benefit and a real harm, nowhere near statistical significance. And for hemorrhagic stroke, the trend ran the other direction: relative risk 2.14 (95% CI 0.96–4.77, p=0.06), a result that missed the conventional significance threshold by a hair but described a nearly doubled risk. One trial, three outcomes, three different relationships between "how confident are we this isn't chance" and "how big is the effect." A p-value attached to any one of these numbers, reported alone, would tell you nothing about the other two.
A "significant" finding that still meant treating hundreds of people
The 2020 Cochrane systematic review of 86 omega-3 fatty acid trials and 162,796 participants makes the same point at a different scale. For coronary heart disease death specifically, the pooled relative risk was 0.90 (95% CI 0.81–1.00; 127,378 participants; 24 trials) — a result the review classified as statistically significant, since the confidence interval's upper bound landed exactly at the no-effect line. The review's authors translated that relative risk into a number needed to treat (NNT) of 334: 334 people would need to take omega-3 supplements, for the duration studied, to prevent one coronary death. That's the effect size made concrete — a small, real benefit spread thin across a large number of people, which is a very different thing to act on than the aspirin trial's 44% relative reduction in heart attacks.
Meanwhile, the same review's estimate for all-cause mortality — relative risk 0.97 (95% CI 0.93–1.01), based on 143,693 participants and rated high-certainty evidence — did not cross the significance threshold at all, despite being calculated from a far larger sample than the "significant" coronary finding. That's not a contradiction. It reflects two separate effect sizes, one slightly larger and one slightly smaller than the point where a 95% confidence interval happens to cross 1.0. Both are small. The label "significant" attached to only one of them.
Why sample size changes the p-value without changing the effect
As a trial or pooled review grows larger, the precision of its estimate improves — the confidence interval narrows around whatever the true effect size actually is. That means a large enough sample can turn even a trivial real difference into a "statistically significant" result, and a small enough sample can leave a genuinely large effect looking non-significant simply because the confidence interval is too wide to rule out zero. Significance testing is sensitive to sample size in a way that effect size, by design, is not. That's precisely why a relative risk, a mean difference, or a number needed to treat — reported with its own confidence interval — carries more decision-relevant information than a p-value or a "significant/not significant" label standing alone.
What we don't know yet
Why the Physicians' Health Study's stroke signal didn't reach significance. A relative risk of 2.14 with a p-value of 0.06 is not proof of harm, but it is not proof of safety either — the trial was not large enough on that specific outcome to resolve the question either way.
Whether a number needed to treat of 334 reflects a stable population-wide effect or conceals more meaningful benefit in a narrower subgroup. The retrieved review pools broad populations; it does not establish whether some individuals see a much larger effect than the average reported here.
How much of the omega-3 coronary finding would survive yet another decade of new trials. A relative risk with a confidence interval landing exactly at the boundary of significance is, by definition, an estimate sitting right at the edge — more data could easily move it either direction.
Whether clinical guidelines that cite a "statistically significant" finding routinely report the underlying effect size alongside it. Neither retrieved study speaks to how these numbers get communicated downstream, only to what the trials themselves found.
Sources
- Physicians' Health Study Steering Committee. Final report on the aspirin component of the ongoing Physicians' Health Study. New England Journal of Medicine, 1989;321(3):129-135. DOI
- Abdelhamid AS, Brown TJ, Brainard JS, et al. Omega-3 fatty acids for the primary and secondary prevention of cardiovascular disease. Cochrane Database of Systematic Reviews, 2020;3(3):CD003177. DOI
Article records maintained via PubMed. Peptide Proof reports what studies found. It does not provide medical advice, dosing guidance, or treatment recommendations.
Sources
Get the evidence, not the hype.
New research briefs, plainly written and fully sourced. No dosing advice, ever.