Testing and interpreting replication studies: Insights from confidence intervals and Bayesian analyses

Abstract

Testing and interpreting replications typically focus on whether they reproduce the sign (direction) and the statistical significance of the parameter estimates reported in a focal study. These side-by-side comparisons of replication and original study findings often yield a dichotomous “successful” or “unsuccessful” replication outcome. However, such practices can lead to incorrect conclusions, lose information by failing to reveal how and to what degree replication findings differ from those of the original research, and threaten replication’s role in safeguarding and advancing empirical literatures. This paper presents two approaches—a multi-staged set of confidence interval tests and Bayesian analyses—which use findings from original and replication studies as inputs for additional assessments of replicability. These methods overcome the limitations of side-by-side comparisons and provide more complete insights into the meaning and boundaries of the original study’s findings. Guidelines for their use are provided, and an empirical example is reported to provide concrete step-by-step illustrations.