Research brief · 2026-09-09
What Is an Effect Size, and Why Does It Matter More Than the P-Value?
A p-value and an effect size answer two different questions. The p-value asks: if this treatment did nothing, how likely would a result this extreme be, just from chance? The effect size asks: how much did the outcome actually change? A relative risk, a mean difference or a number needed to treat answers that directly. A p-value can be tiny because an effect is large, or because a sample is big enough to detect a trivial difference.
The Physicians' Health Study, in the New England Journal of Medicine in 1989, randomized 22,071 physicians to low-dose aspirin or placebo. For heart attack, the result was unambiguous on both counts: a relative risk of 0.56, a 44 percent reduction, with a p-value below 0.00001. But in the same patients, overall cardiovascular mortality showed a relative risk of 0.96 with an interval wide enough to include benefit and harm, and hemorrhagic stroke ran the other way, a relative risk of 2.14 that just missed significance at p equals 0.06 while describing a nearly doubled risk.
The 2020 Cochrane review by Abdelhamid and colleagues makes the point at the other scale. Pooling 86 omega-3 trials and 162,796 participants, the relative risk for coronary heart disease death was 0.90, with the interval's upper bound landing exactly on the no-effect line. The review called it significant, and translated it to a number needed to treat of 334: 334 people treated for the studied duration to prevent one coronary death. A small, real benefit spread thin. All-cause mortality, from an even larger sample, did not cross the threshold.
Sample size changes the p-value without changing the effect, which is why an effect size with its own confidence interval carries more decision-relevant information than a significant-or-not label. What these studies do not settle is whether the aspirin stroke signal was real harm or chance, or whether an NNT of 334 hides a larger benefit in some narrower group. Aspirin carries its own risks, and any decision about it or any supplement belongs with a physician. Sources are listed below.
Sources
- Physicians' Health Study Steering Committee, Final report on the aspirin component of the ongoing Physicians' Health Study (n=22,071)
- New England Journal of Medicine, 1989
- Abdelhamid AS et al., Omega-3 fatty acids for the primary and secondary prevention of cardiovascular disease (86 trials, n=162,796)
- Cochrane Database of Systematic Reviews, 2020
Peptide Proof reports what studies found. We do not give medical advice, dosing guidance, or treatment recommendations.