Tl;dr: (1) Expected false positive rate for any one statistical test is generally 0.05 (aka, alpha). But this error compounds when you run multiple statistical tests. (2) Adjust your target p-value by applying the Bonferonni correction (0.05/n where n = # statistical to tests) to see if authors’ findings are truly consistent with their reported findings.
Sometimes stats are really boring and awful. Sometimes they can be great and truly deepen your understanding of a paper. Hopefully this discussion on this underutilized stat will fall into the latter.
The problem of multiple comparisons is common in literature and often unrecognized by authors and readers alike. Every statistical test inherently has a false positive rate; we generally accept alpha = 0.05 as an acceptable false positive rate (hence the obsession with p<0.05). If you run multiple tests, then the generation of a false positive becomes more and more likely. Authors generally work around this issue by proclaiming one primary study outcome and then a litany of secondary outcomes. It isn’t a horrible strategy if you force yourself to only interpret the p-values on the primary study outcome (which we suck at doing). It is a horrible strategy, mainly because the underlining math hasn’t changed at all, when interpreting secondary study outcomes.
If you want to try to draw more meaningful conclusions from secondary study outcomes, then consider applying the Bonferroni correction. It is a simple correction of dividing your desired false positive rate (generally alpha = 0.05) by the number of outcomes tested (n): 0.05 / n. For example, if the authors ran two different statistical tests with separate outcomes, then the desired p-value should be <0.025 (or 0.05/2).
Know that the Bonferonni correction is an overly conservative assumption, but it is easy to understand and apply quickly while reviewing a paper. Consider applying it as you try to draw more meaningful conclusions from secondary analyses in papers.
Source:
Sedgwick, P. Multiple hypothesis testing and Bonferroni’s correction. BMJ 2014; 349 doi: https://doi.org/10.1136/bmj.g6284
