This article introduces a new approach for evaluating replication results. It combines effect-size estimation with hypothesis testing, assessing the extent to which the replication results are consistent with an effect size big enough to have been detectable in the original study. The approach is demonstrated by examining replications of three well-known findings. Its benefits include the following: (a) differentiating “unsuccessful” replication attempts (i.e., studies yielding p > .05) that are too noisy from those that actively indicate the effect is undetectably different from zero, (b) “protecting” true findings from underpowered replications, and (c) arriving at intuitively compelling inferences in general and for the revisited replications in particular.
AlognaV. K.AttayaM. K.AucoinP.BahníkŠ.BirchS.BirtA. R.. . . ZwaanR. A. (2014). Registered Replication Report: Schooler and Engstler-Schooler (1990). Perspectives on Psychological Science, 9, 556–578.
2.
AsendorpfJ. B.ConnerM.De FruytF.De HouwerJ.DenissenJ. J.FiedlerK.. . . WichertsJ. M. (2013). Recommendations for increasing replicability in psychology. European Journal of Personality, 27, 108–119.
BondR. M.FarissC. J.JonesJ. J.KramerA. D.MarlowC.SettleJ. E.FowlerJ. H. (2012). A 61-million-person experiment in social influence and political mobilization. Nature, 489, 295–298.
5.
ButtonK. S.IoannidisJ. P.MokryszC.NosekB. A.FlintJ.RobinsonE. S.MunafòM. R. (2013). Power failure: Why small sample size undermines the reliability of neuroscience. Nature Reviews Neuroscience, 14, 365–376.
6.
CohenJ. (1962). The statistical power of abnormal-social psychological research: A review. Journal of Abnormal and Social Psychology, 65, 145–153.
7.
CohenJ. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Erlbaum.
8.
CohenJ. (1994). The earth is round (p < .05). American Psychologist, 49, 997–1003.
9.
CourseyD.HovisJ.SchulzeW. (1987). The disparity between willingness to accept and willingness to pay measures of value. Quarterly Journal of Economics, 102, 679–690.
10.
CummingG. (2008). Replication and p intervals: p values predict the future only vaguely, but confidence intervals do much better. Perspectives on Psychological Science, 3, 286–300.
11.
CummingG. (2014). The new statistics: Why and how. Psychological Science, 25, 7–29.
12.
DahlM. S.DezsāC. L.RossD. G. (2012). Fatherhood and managerial style: How a male CEO’s children affect the wages of his employees. Administrative Science Quarterly, 57, 669–693.
FisherR. A. (1926). The arrangement of field experiments. Journal of the Ministry of Agriculture, 33, 503–513.
15.
GámezE.DíazJ. M.MarreroH. (2011). The uncertain universality of the Macbeth effect with a Spanish sample. The Spanish Journal of Psychology, 14, 156–162.
16.
GreenwaldA. G. (1975). Consequences of prejudice against the null hypothesis. Psychological Bulletin, 82, 1–20.
17.
HedgesL. V. (1984). Estimation of effect size under nonrandom sampling: The effects of censoring studies yielding statistically insignificant mean differences. Journal of Educational and Behavioral Statistics, 9, 61–85.
18.
HodgesJ.LehmannE. (1954). Testing the approximate validity of statistical hypotheses. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 16, 261–268.
19.
IoannidisJ. P. A. (2008). Why most discovered true associations are inflated. Epidemiology, 19, 640–646.
20.
JeffreysH. (1961). Theory of probability (3rd ed.). Oxford, England: Oxford University Press.
21.
KahnemanD.KnetschJ. L.ThalerR. H. (1990). Experimental tests of the endowment effect and the Coase theorem. Journal of Political Economy, 98, 1325–1348.
22.
KnetschJ. L.SindenJ. A. (1984). Willingness to pay and compensation demanded: Experimental evidence of an unexpected disparity in measures of value. Quarterly Journal of Economics, 99, 507–521.
23.
KraemerH. C. (1983). Theory of estimation and testing of effect sizes: Use in meta-analysis. Journal of Educational and Behavioral Statistics, 8, 93–101.
24.
LaneD. M.DunlapW. P. (1978). Estimating effect size: Bias resulting from the significance criterion in editorial decisions. British Journal of Mathematical and Statistical Psychology, 31, 107–112.
25.
LindleyD. V. (1965). Introduction to probability and statistics from a Bayesian viewpoint (Vol. 2). Cambridge, England: Cambridge University Press.
26.
LucasR. E.LawlessN. M. (2013). Does life seem better on a sunny day? Examining the association between daily weather conditions and life satisfaction judgments. Journal of Personality and Social Psychology, 104, 872–884.
27.
NosekB. A.BanajiM.GreenwaldA. G. (2002). Harvesting implicit group attitudes and beliefs from a demonstration web site. Group Dynamics: Theory, Research, and Practice, 6, 101–115.
28.
Open Science Collaboration. (2012). An open, large-scale, collaborative effort to estimate the reproducibility of psychological science. Perspectives on Psychological Science, 7, 657–660.
RindskopfD. M. (1997). Testing ‘small’, not null, hypotheses: Classical and Bayesian approaches. In HarlowL. L.MulaikS. A.SteigerJ. H. (Eds.), What if there were no significance tests (pp. 319–332). Mahwah, NJ: Erlbaum.
31.
RossiJ. S. (1990). Statistical power of psychological research: What have we gained in 20 years?Journal of Consulting and Clinical Psychology, 58, 646–656.
32.
RouanetH. (1996). Bayesian methods for assessing importance of effects. Psychological Bulletin, 119, 149–158.
33.
RouderJ. N.SpeckmanP. L.SunD.MoreyR. D.IversonG. (2009). Bayesian t tests for accepting and rejecting the null hypothesis. Psychonomic Bulletin & Review, 16, 225–237.
34.
SchoolerJ. W.Engstler-SchoolerT. Y. (1990). Verbal overshadowing of visual memories: Some things are better left unsaid. Cognitive Psychology, 22, 36–71.
35.
SchwarzN.CloreG. (1983). Mood, misattribution, and judgments of well-being: Informative and directive functions of affective states. Journal of Personality and Social Psychology, 45, 513–523.
36.
SedlmeierP.GigerenzerG. (1989). Do studies of statistical power have an effect on the power of studies?Psychological Bulletin, 105, 309–316.
37.
SerlinR. C.LapsleyD. K. (1985). Rationality in psychological research: The good-enough principle. American Psychologist, 40, 73–83.
38.
SerlinR. C.LapsleyD. K. (1993). Rational appraisal of psychological research and the good-enough principle. In KerenG.LewisC. (Eds.), A handbook for data analysis in the behavioral sciences: Methodological issues (pp. 199–228). Mahwah, NJ: Erlbaum.
39.
SievJ. (2012). [Attempt to replicate Zhong & Liljenquist]. Unpublished raw data.
SimonsohnU.NelsonL. D.SimmonsJ. P. (2014a). p-curve and effect size: Correcting for publication bias using only significant results. Perspectives on Psychological Science, 9, 666–681.
42.
SimonsohnU.NelsonL. D.SimmonsJ. P. (2014b). P-curve: A key to the file-drawer. Journal of Experimental Psychology: General, 143, 534–547.
43.
ValentineJ. C.BiglanA.BoruchR. F.CastroF. G.CollinsL. M.FlayB. R.. . . SchinkeS. P. (2011). Replication in prevention science. Prevention Science, 12, 103–117.
44.
VerhagenJ.WagenmakersE.-J. (2014). A Bayesian test to quantify the success or failure of a replication attempt. Journal of Experimental Psychology: General, 143, 1457–1475.
45.
ZhongC.-B.LiljenquistK. (2006). Washing away your sins: Threatened morality and physical cleansing. Science, 313, 1451–1452.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
