Research brief · 2026-09-09
What 'Statistically Significant' Actually Means
Statistically significant is one of the most quoted and least understood phrases in health news. A p-value answers one narrow question: if the treatment truly did nothing, how likely would a gap this large appear by chance alone? It does not mean the treatment is 99 percent likely to work, and it is not a measure of how big the benefit is.
The SELECT trial, in the New England Journal of Medicine in 2023, is a clear case. It randomized 17,604 adults with cardiovascular disease and overweight or obesity to semaglutide or placebo. A major cardiovascular event occurred in 6.5 percent of the semaglutide group and 8.0 percent of the placebo group, hazard ratio 0.80, P less than 0.001. A gap that size in a trial that large would be very unlikely to arise by chance.
A 2026 secondary analysis in Nature Medicine by Meyhofer and colleagues shows why size and certainty are different things. In subgroups at higher risk of liver fibrosis, the broadest showed a 26 percent reduction in events, significant at P equals 0.0004. The narrowest showed the biggest reduction, 34 percent, yet its confidence interval ran from 0.39 to 1.10 and the result was not significant, P equals 0.11. The most dramatic number came from the smallest group, and the smallest group gave the least precise estimate.
That is how the statistics are supposed to work. Confidence intervals widen as sample size shrinks. A hazard ratio of 0.66 sounds better than 0.74, but its interval could not rule out anything from a 61 percent reduction to a 10 percent increase, while every value in the 0.74 interval pointed toward benefit.
The 0.05 cutoff is a convention, not a law of nature. Nothing biologically separates P equals 0.04 from 0.06. What none of this tells you is whether an effect is large enough to matter for a given person, and testing many subgroups raises the odds that one looks significant by chance. Semaglutide is a prescription medication, and decisions about it belong with a physician. Sources are listed below.
Sources
- Lincoff AM et al., Semaglutide and cardiovascular outcomes in obesity without diabetes (SELECT trial, n=17,604)
- New England Journal of Medicine, 2023
- Meyhofer SM et al., Semaglutide on liver fibrosis and heart outcomes in patients at high risk of liver fibrosis: prespecified analysis of SELECT
- Nature Medicine, 2026
Peptide Proof reports what studies found. We do not give medical advice, dosing guidance, or treatment recommendations.