Statistical power analysis provides the conventional approach to assess error rates when designing a research study. However, power analysis is flawed in that a narrow emphasis on statistical significance is placed as the primary focus of study design. In noisy, small-sample settings, statistically significant results can often be misleading. To help researchers address this problem in the context of their own studies, we recommend design calculations in which (a) the probability of an estimate being in the wrong direction (Type S [sign] error) and (b) the factor by which the magnitude of an effect might be overestimated (Type M [magnitude] error or exaggeration ratio) are estimated. We illustrate with examples from recent published research and discuss the largest challenge in a design calculation: coming up with reasonable estimates of plausible effect sizes based on external information.
BroerL.LillC. M.SchuurM.AminN.RoehrJ. T.BertramL.. . . van DuijnC. M. (2013). Distinguishing true from false positives in genomic studies: P values. European Journal of Epidemiology, 28, 131–138.
2.
ButtonK. S.IoannidisJ. P. A.MokryszC.NosekB.FlintJ.RobinsonE. S. J.MunafoM. R. (2013). Power failure: Why small sample size undermines the reliability of neuroscience. Nature Reviews Neuroscience, 14, 1–12.
3.
ChenY.EbensteinA.GreenstoneM.LiH. (2013). Evidence on the impact of sustained exposure to air pollution on life expectancy from China’s Huai River policy. Proceedings of the National Academy of Sciences, USA, 110, 12936–12941.
4.
CohenJ. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Mahwah, NJ: Erlbaum.
5.
DuranteK.ArsenaA.GriskeviciusV. (2013). The fluctuating female vote: Politics, religion, and the ovulatory cycle. Psychological Science, 24, 1007–1016.
6.
FinkelS. E. (1993). Reexamining the “minimal effects” model in recent presidential campaign. Journal of Politics, 55, 1–21.
7.
FroehlichG. W. (1999). What is the chance that this study is clinically significant? A proposal for Q values. Effective Clinical Practice, 2, 234–239.
GanleyC. M.MingleL. A.RyanA. M.RyanK.VasilyevaM.PerryM. (2013). An examination of stereotype threat effects on girls’ mathematics performance. Developmental Psychology, 49, 1886–1897.
10.
GelmanA. (2007). Letter to the editor regarding some papers of Dr. Satoshi Kanazawa. Journal of Theoretical Biology, 245, 597–599.
11.
GelmanA. (2013). P values and statistical practice. Epidemiology, 24, 69–72.
12.
GelmanA. (2014). The connection between varying treatment effects and the crisis of unreplicable research: A Bayesian perspective. Journal of Management. Advance online publication. doi:10.1177/0149206314525208
13.
GelmanA.TuerlinckxF. (2000). Type S error rates for classical and Bayesian single and multiple comparison procedures. Computational Statistics, 15, 373–390.
14.
GelmanA.WeakliemD. (2009). Of beauty, sex, and power: Statistical challenges in the estimation of small effects. American Scientist, 97, 310–316.
15.
GoodmanS. N.BerlinJ. A. (1994). The use of predicted confidence intervals when planning experiments and the misuse of power when interpreting results. Annals of Internal Medicine, 121, 200–206.
16.
HedgesL. V. (1984). Estimation of effect size under nonrandom sampling: The effects of censoring studies yielding statistically insignificant mean differences. Journal of Educational Statistics, 9, 61–85.
17.
HillygusD. S.JackmanS. (2003). Voter decision making in election 2000: Campaign effects, partisan activation, and the Clinton legacy. American Journal of Political Science, 47, 583–596.
18.
HoenigJ. M.HeiseyD. M. (2001). The abuse of power: The pervasive fallacy of power calculations for data analysis. American Statistician, 55, 1–6.
19.
KanazawaS. (2007). Beautiful parents have more daughters: A further implication of the generalized Trivers-Willard hypothesis. Journal of Theoretical Biology, 244, 133–140.
20.
LaneD. M.DunlapW. P. (1978). Estimating effect size: Bias resulting from the significance criterion in editorial decisions. British Journal of Mathematical and Statistical Psychology, 31, 107–112.
21.
LehrerJ. (2010, December13). The truth wears off. New Yorker, pp. 52–57.
22.
LenthR. V. (2007). Statistical power calculations. Journal of Animal Science, 85, E24–E29.
23.
NosekB. A.SpiesJ. R.MotylM. (2013). Scientific utopia: II. Restructuring incentives and practices to promote truth over publishability. Perspectives on Psychological Science, 7, 615–631.
24.
SchulzK. F.GrimesD. A. (2005). Sample size calculations in randomised trials: Mandatory and mystical. Lancet, 365, 1348–1353.
25.
SennS. J. (2002). Power is indeed irrelevant in interpreting completed studies. British Medical Journal, 325, Article 1304. doi:10.1136/bmj.325.7375.1304
26.
SimmonsJ.NelsonL.SimonsohnU. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allow presenting anything as significant. Psychological Science, 22, 1359–1366.
27.
SterneJ. A.SmithG. D. (2001). Sifting the evidence—What’s wrong with significance tests?British Medical Journal, 322, 226–231.
28.
StreppelM. T.BoshuizenH. C.OckeM. C.KokF. J.KromhoutD. (2007). Mortality and life expectancy in relation to long-term cigarette, cigar and pipe smoking: The Zutphen Study. Tobacco Control, 16, 107–113.
29.
TukeyJ. W. (1977). Exploratory data analysis. New York, NY: Addison-Wesley.
30.
VeehofM. M.OskamM.-J.SchreursK. M. G.BohlmeijerE. T. (2011). Acceptance-based interventions for the treatment of chronic pain: A systematic review and meta-analysis. Pain, 152, 533–542.
31.
VulE.HarrisC.WinkelmanP.PashlerH. (2009). Puzzlingly high correlations in fMRI studies of emotion, personality, and social cognition (with discussion). Perspectives on Psychological Science, 4, 274–290.
