Results Provide Information About Cumulative Probabilities of Finding Significance: Let’s Report This Information

Abstract

Commonly reported statistics, such as the t value and p value, contain useful information about the cumulative probability of finding statistical significance based on the properties of the sample being analyzed. Unfortunately, converting t values and p values into this form of information is not intuitive and is often done incorrectly. We show how the bootstrap can provide a way to understand the cumulative probability of finding significance based on the characteristics of a specific sample and the statistical model being used. We also provide a simple way to estimate this probability from t values without having to rely on the bootstrap. Reporting sample-based probabilities can help promote robust and reliable research by conveying appropriate levels of uncertainty into discussions of results. We provide recommendations to authors, editors, reviewers, readers, and educators to help counter origination bias (i.e., how much a single-study finding should be viewed as solid or sacred) and other biases tied to misunderstanding the variability inherent associated with reporting “statistically significant” findings.