A result is called statistically significant when its p-value falls below a threshold set in advance, often 0.05, and the American Statistical Association’s president said in 2016 that the p-value had become a “gatekeeper” for publication in some fields (ASA, 2016). The p-value describes how incompatible the data are with a model in which there is no real difference. It does not give the probability that a hypothesis is true (ASA, 2016).
Psychedelic trials show why the word needs care. In a 59-person trial, psilocybin and escitalopram changed a self-rated depression score by −8.0 and −6.0, and the difference was not significant (P = 0.17) (Carhart-Harris et al., 2021). Secondary outcomes favoured psilocybin, but were not corrected for testing many outcomes at once (Carhart-Harris et al., 2021). In the 2026 EPISODE trial, the first planned comparison was not significant, so formal testing stopped (Mertens et al., 2026).
In 2016 the American Statistical Association warned that decisions should not rest only on whether a p-value passes a threshold, and that significance does not measure the size or importance of a result (ASA, 2016). That is why journal summaries report effect sizes and confidence intervals alongside p-values.
Sources
- ASA statement on statistical significance and p-values, 7 March 2016
- Carhart-Harris et al., NEJM, 2021
- Mertens et al., JAMA Psychiatry, 2026
Last checked .