Issues surrounding the importance and interpretation of replication research have generated considerable debate and controversy in recent years. Some of the controversy can be attributed to imprecise and inadequate specifications of the statistical criteria needed to assess replication and nonreplication. Two types of statistical replication evidence and four types of statistical nonreplication evidence are described. In addition, three types of inconclusive statistical replication evidence are described. An important benefit of a replication study is the ability to combine an effect-size estimate from the original study with an effect-size estimate from the follow-up study to obtain a more precise and generalizable effect-size estimate. The sample size in the follow-up study is an important design consideration, and some methods for determining the follow-up sample size requirements are discussed. R functions are provided that can be used to analyze results from a replication study. R functions to determine the appropriate sample size in the follow-up study also are provided.
AndersonS. F.MaxwellS. E. (2016). There’s more than one way to conduct a replication study: Beyond statistical significance. Psychological Methods, 21, 1–12.
2.
BardiA.ZentnerM. (2017). Grand challenges for personality and social psychology: Moving beyond the replication crisis. Frontiers in Psychology, 8, Article 2068. https://doi.org/10.3389/fpsyg.2017.02068
3.
BonettD. G. (2012). Replication-extension studies. Current Directions in Psychology, 21, 409–412.
4.
BonettD. G.PriceR. M. (2015). Varying coefficient meta-analysis methods for odds ratios and risk ratios. Psychological Methods, 20, 394–406.
5.
BonettD. G.WrightT. A. (2007). Comments and recommendations regarding the hypothesis testing controversy. Journal of Organizational Behavior28, 647–659.
6.
BorensteinM.HedgesL. V.HigginsJ. P. T.RothsteinH. R. (2009). Introduction to meta-analysis. Wiley.
7.
ButlerN.DelaneyH.SpoelstraS. (2017). The gray zone: Questionable research practices in the business school. Academy of Management Learning & Education, 16, 94–109.
8.
CamererC. F.DreberA.ForsellE.HoT. H.HuberJ.JohannessonM.KirchlerM.AlmenbergJ.AltmejdA.ChanT.HeikenstenE.HolzmeisterF.ImaiT.IsakssonS.NaveG.PfeifferT.RazenM.WuH. (2016). Evaluating replicability of laboratory experiments in economics. Science, 351, 1433–1436.
9.
ChangA. C.LiP. (2015). Is economics research replicable? Sixty published papers from thirteen journals say “Usually Not”(Finance and Economics Discussion Series 2015-083). Board of Governors of the Federal Reserve System. https://doi.org/10.17016/FEDS.2015.083
10.
ChenH.CohenP.ChenS. (2010). How big is a big odds ratio? Interpreting the magnitudes of odds ratios in epidemiological studies. Communications in Statistics – Simulation and Computation, 39, 860–864.
11.
CohenJ. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Erlbaum.
12.
CoyneJ. C. (2016). Replication initiatives will not salvage the trustworthiness of psychology. BMC Psychology, 4, Article 28. https://doi.org/10.1186/s40359-016-0134-3
13.
EbersoleC. R.AthertonO. E.BelangerA. L.SkulborstadH. M.AllenJ. M.BanksJ. B.BaranskiE.BernsteinM. J.BonfiglioD. B. V.BoucherL.BrownE. R.BudimanN. I.CairoA. H.CapaldiC. A.ChartierC. R.ChungJ. M.CiceroD. C.ColemanJ. A.ConwayJ. G.…NosekB. A. (2016). Many Labs 3. Evaluating participant pool quality across the academic semester via replication. Journal of Experimental Social Psychology, 67, 68–82.
14.
FuchsH. M.JennyM.FiedlerS. (2012). Psychologists are open to change, yet wary of rules. Perspectives on Psychological Science, 7, 639–642.
15.
GelmanA.CarlinJ. (2014). Beyond power calculations: Assessing Type S (Sign) and Type M (Magnitude) errors. Perspectives on Psychological Science, 9, 641–651.
16.
GelmanA.SternH. (2006). The difference between “significant” and “not significant” is not itself statistically significant. The American Statistician, 60, 328–331.
17.
GilbertD. T.KingG.PettigrewS.WilsonT. D. (2016). Comment on “Estimating the reproducibility of psychological science.”Science, 357, 1037a–1038a.
18.
HowellD. C. (2012). Statistical methods for psychology (8th ed.). Wadsworth.
19.
IoannidisJ. P. A. (2016). Anticipating consequences of sharing raw data and code and of awarding badges for sharing. Journal of Clinical Epidemiology, 70, 258–260.
20.
JanzN. (2016). Bringing the gold standard into the classroom: Replication in university teaching. International Studies Perspectives, 17, 392–407.
21.
JonesL. V.TukeyJ. W. (2000). A sensible formulation of the significance test. Psychological Methods, 5, 411–414.
22.
KeppelG.WickensT. D. (2004). Design and analysis: A researcher’s handbook (4th ed.). Pearson.
23.
KleinR. A.VianelloM.HasselmanF.AdamsB. G.AdamsR. B.JrAlperS.AveyardM.AxtJ. R.BabalolaM. T.BahníkS.BatraR.BerkicsM.BernsteinM. J.BerryD. RBialobrzeskaO.BinanE. DBocianK.BrandtM. JBuschingR.RédeiA. C. (2018). Many Labs 2. Investigating variation in replicability across sample and setting. Advances in Methods and Practices in Psychological Science, 1, 443–490.
24.
KlineR. B. (2019). Becoming a behavioral science researcher (2nd ed.). Guilford Press.
25.
KöhlerT.CortinaJ. M. (in press). Play it again, Sam! An analysis of constructive replication in the organizational sciences. Journal of Management.
26.
LakensD. (2017). Equivalence tests: A practical primer for t tests, correlation, and meta- analyses. Social Psychological and Personality Science, 8, 355–362.
MathewsP. (2010). Sample size calculations: Practical methods for engineers and scientists. Mathews Malnar and Bailey, Inc.
29.
McClaveJ.BensonP. G.SincichT. T. (2012). Statistics for business and economics (12th ed.). Pearson.
30.
NeuliepJ. W.CrandallR. (1993). Reviewer bias against replication research. Journal of Social Behavior and Personality, 8, 21–29.
31.
NosekB. A.LakensD. (2014). Registered reports: A method to increase the credibility of published results. Social Psychology, 45, 137–141.
32.
Open Science Collaboration. (2015). Estimating the reproducibility of psychological studies. Science, 349, aac4716.
33.
PrinzF.SchlangeT.AsadullahK. (2011). Believe it or not: How much can we rely on published data on potential drug targets?Nature Reviews Drug Discovery, 10, 712.
34.
RosenthalR.RosnowR. L. (1984). Essentials in behavioral research: Methods and data analysis. McGraw-Hill.
SchmidtF. L.HunterJ. E. (2015). Methods of meta-analysis: Correcting for error and bias in research findings. Sage.
37.
StandingL. G.GrenierM.LaneE. A.RobertsM. S.SykesS. J. (2014). Using replication projects in teaching research methods. Psychology Teaching Review, 20, 96–104.
38.
TsangE. W. K.KwanK-M. (1999). Replication and theory development in organization science: A critical realistic perspective. Academy of Management Review, 24, 759–780.
39.
WeissM.MorrisonE. W. (2019). Speaking up and moving up: How voice can enhance employees’ social status. Journal of Organizational Behavior, 40, 5–19.
40.
WellekS. (2010). Testing statistical hypotheses of equivalence and noninferiority (2nd ed.). Chapman & Hall.
41.
ZwaanR. A.EtzA.LucasR. E.DonnellanM. B. (2018). Making replication mainstream. Behavioral and Brain Sciences, 41, e120.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
