An essential first step in planning a confirmatory or a replication study is to determine the sample size necessary to draw statistically reliable inferences using power analysis. A key problem, however, is that what is available is the sample-size estimate of the effect size, and its use can lead to severely underpowered studies when the effect size is overestimated. As a potential remedy, we introduce safeguard power analysis, which uses the uncertainty in the estimate of the effect size to achieve a better likelihood of correctly identifying the population effect size. Using a lower-bound estimate of the effect size, in turn, allows researchers to calculate a sample size for a replication study that helps protect it from being underpowered. We show that in most common instances, compared with nominal power, safeguard power is higher whereas standard power is lower. We additionally recommend the use of safeguard power analysis to evaluate the strength of the evidence provided by the original study.
AsendorpfJ. B.ConnerM.De FruytF.De HouwerJ.DenissenJ. J. A.FiedlerK.. . . WichertsJ. M. (2013). Recommendations for increasing replicability in psychology. European Journal of Personality, 27, 108–119. doi:10.1002/per.1919
2.
BakkerM.van DijkA.WichertsJ. M. (2012). The rules of the game called psychological science. Perspectives on Psychological Science, 7, 543–554. doi:10.1177/1745691612459060
3.
BrandtM. J.IjzermanH.DijksterhuisA.FarachF. J.GellerJ.Giner-SorollaR.. . . van ’t VeerA. (2014). The replication recipe: What makes for a convincing replication?Journal of Experimental Social Psychology, 50, 217–224. doi:10.1016/j.jesp.2013.10.005
4.
BraverS. L.ThoemmesF. J.RosenthalR. (2014). Continuously cumulating meta-analysis and replicability. Perspectives on Psychological Science, 9, 333–342.
5.
ButtonK. S.IoannidisJ. P. A.MokryszC.NosekB. A.FlintJ.RobinsonE. S. J.MunafòM. R. (2013). Power failure: Why small sample size undermines the reliability of neuroscience. Nature Reviews Neuroscience, 14, 365–376. doi:10.1038/nrn3475
6.
CohenJ. (1962). The statistical power of abnormal-social psychological research: A review. Journal of Abnormal and Social Psychology, 65, 145–153. doi:10.1037/h0045186
7.
CohenJ. (1988). Statistical power analysis for the behavioral sciences. Hillsdale, NJ: Erlbaum.
8.
CohenJ. (1992). A power primer. Psychological Bulletin, 112, 155–159. doi:10.1037/0033-2909.112.1.155
9.
CummingG. (2012). Understanding the new statistics: Effect sizes, confidence intervals, and meta-analysis. New York, NY: Routledge.
10.
CummingG.FidlerF. (2010). Effect sizes and confidence intervals. In HancockG. R.MuellerR. O. (Eds.), The reviewer’s guide to quantitative methods in the social sciences (pp. 79–91). London, England: Routledge.
FanelliD.IoannidisJ. P. A. (2013). US studies may overestimate effect sizes in softer research. Proceedings of the National Academy of Sciences, USA, 110, 15031–15036. doi:10.1073/pnas.1302997110
14.
FergusonC. J.BrannickM. T. (2012). Publication bias in psychological science: Prevalence, methods for identifying and controlling, and implications for the use of meta-analyses. Psychological Methods, 17, 120–128. doi:10.1037/a0024445
15.
FrickR. W. (1996). The appropriate use of null hypothesis testing. Psychological Methods, 1, 379–390. doi:10.1037//1082–989X.1.4.379
16.
GalakJ.LeBoeufR. A.NelsonL. D.SimmonsJ. P. (2012). Correcting the past: Failures to replicate psi. Journal of Personality and Social Psychology, 103, 933–948. doi:10.1037/a0029709
17.
GigerenzerG.KraussS.VitouchO. (2004). The null ritual: What you always wanted to know about significance testing but were afraid to ask. In KaplanD. (Ed.), The SAGE handbook of quantitative methodology for the social sciences (pp. 391–408). Thousand Oaks, CA: Sage.
18.
HowellD. C. (2002). Statistical methods for psychology (5th ed.). Pacific Grove, CA: Duxbury.
19.
JohnL. K.LoewensteinG.PrelecD. (2012). Measuring the prevalence of questionable research practices with incentives for truth telling. Psychological Science, 23, 524–532. doi:10.1177/0956797611430953
20.
KelleyK.MaxwellS. E. (2003). Sample size for multiple regression: Obtaining regression coefficients that are accurate, not simply significant. Psychological Methods, 8, 305–321. doi:10.1037/1082-989X.8.3.305
21.
KelleyK.RauschJ. R. (2006). Sample size planning for the standardized mean difference: Accuracy in parameter estimation via narrow confidence intervals. Psychological Methods, 11, 363–385. doi:10.1037/1082-989X.11.4.363
22.
KievitR. A. (2011). Bayesians caught smuggling priors into Rotterdam harbor. Perspectives on Psychological Science, 6, 313. doi:10.1177/1745691611406928
23.
KruschkeJ. K. (2010). What to believe: Bayesian methods for data analysis. Trends in Cognitive Sciences, 14, 293–300. doi:10.1016/j.tics.2010.05.001
24.
KruschkeJ. K. (2011). Bayesian assessment of null values via parameter estimation and model comparison. Perspectives on Psychological Science, 6, 299–312. doi:10.1177/1745691611406925
25.
LakensD. (2014). Sailing from the seas of chaos into the corridor of stability: Practical recommendations to increase the informational value of studies. Perspectives on Psychological Science, 9, 278–292.
26.
LeBelE. P.CampbellL. (2013). Heightened sensitivity to temperature cues in individuals with high anxious attachment: Real or elusive phenomenon?Psychological Science, 24, 2128–2130. doi:10.1177/0956797613486983
27.
LedgerwoodA.ShroutP. E. (2011). The trade-off between accuracy and precision in latent variable models of mediation processes. Journal of Personality and Social Psychology, 101, 1174–1188.
28.
MasicampoE. J.LalandeD. R. (2012). A peculiar prevalence of p values just below .05. Quarterly Journal of Experimental Psychology, 65, 2271–2279. doi:10.1080/17470218.2012.711335
29.
MaxwellS. E.KelleyK.RauschJ. R. (2008). Sample size planning for statistical power and accuracy in parameter estimation. Annual Review of Psychology, 59, 537–563. doi:10.1146/annurev.psych.59.103006.093735
30.
NickersonR. S. (2000). Null hypothesis significance testing: A review of an old and continuing controversy. Psychological Methods, 5, 241–301. doi:10.1037/1082-989X.5.2.241
31.
NosekB. A.SpiesJ. R.MotylM. (2012). Scientific utopia: II. Restructuring incentives and practices to promote truth over publishability. Perspectives on Psychological Science, 7, 615–631.
32.
Open Science Collaboration. (2012). An open, large-scale, collaborative effort to estimate the reproducibility of psychological science. Perspectives on Psychological Science, 7, 657–660. doi:10.1177/1745691612462588
33.
PashlerH.WagenmakersE. J. (2012). Editors’ introduction to the special section on replicability in psychological Science: A crisis of confidence?Perspectives on Psychological Science, 7, 528–530. doi:10.1177/1745691612465253
34.
RosnowR. L.RosenthalR. (1989). Statistical procedures and the justification of knowledge in psychological science. American Psychologist, 44, 1276–1284. doi:10.1037/0003-066x.44.10.1276
35.
SchönbrodtF. D.PeruginiM. (2013). At what sample size do correlations stabilize?Journal of Research in Personality, 47, 609–612. doi:10.1016/j.jrp.2013.05.009
SimmonsJ. P.NelsonL. D.SimonsohnU. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22, 1359–1366. doi:10.1177/0956797611417632
38.
SimonsohnU. (2013). Small telescopes: Detectability and the evaluation of replication results. Retrieved from http://ssrn.com/abstract=2259879
39.
SterlingT. D.RosenbaumW. L.WeinkamJ. J. (1995). Publication decisions revisited: The effect of the outcome of statistical tests on the decision to publish and vice versa. The American Statistician, 49, 108–112. doi:10.2307/2684823
40.
WagenmakersE. J.WetzelsR.BorsboomD.van der MaasH. L. J. (2011). Why psychologists must change the way they analyze their data: The case of psi: Comment on Bem (2011). Journal of Personality and Social Psychology, 100, 426–432. doi:10.1037/a0022790
41.
WagenmakersE. J.WetzelsR.BorsboomD.van der MaasH. L. J.KievitR. A. (2012). An agenda for purely confirmatory research. Perspectives on Psychological Science, 7, 632–638. doi:10.1177/1745691612463078
WilkinsonL., & Task Force on Statistical Inference. (1999). Statistical methods in psychology journals: Guidelines and explanations. American Psychologist, 54, 594–604. doi:10.1037/0003-066X.54.8.594
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
