This methodological guidance article discusses the elements of a high-quality meta-analysis that is conducted within the context of a systematic review. Meta-analysis, a set of statistical techniques for synthesizing the results of multiple studies, is used when the guiding research question focuses on a quantitative summary of study results. In this guidance article, we discuss the systematic review methods that support high-quality meta-analyses and outline best practice meta-analysis methods for describing the distribution of effect sizes in a set of eligible studies. We also provide suggestions for transparently reporting the methods and results of meta-analyses to influence practice and policy. Given the increasing use of meta-analysis for important policy decisions, the methods and results of meta-analysis should be both transparent and reproducible.
AgrawalS.RaoS. C.BulsaraM. K.PatoleS. K. (2018). Prevalence of autism spectrum disorder in preterm infants: A meta-analysis. Pediatrics, 142(3), e20180134. doi:10.1542/peds.2018-0134
2.
AlexanderP. (in press). The art and science of quality systematic reviews. Review of Educational Research.
3.
AloeA. M.BeckerB. J. (2012). An effect size for regression predictors in meta-analysis. Journal of Educational and Behavioral Statistics, 37, 278–297. doi:10.3102/1076998610396901
4.
AloeA. M.ThompsonC. G. (2013). The synthesis of partial effect sizes. Journal of the Society for Social Work and Research, 4, 390–405. doi:10.5243/jsswr.2013.24
5.
AndersonD.SpybrookJ.MaynardR. (2019). REES: A registry of efficacy and effectiveness studies in education. Educational Researcher, 48, 45–50. doi:10.3102/0013189X18810513
6.
AppelbaumM.CooperH.KlineR. B.Mayo-WilsonE.NezuA. M.RaoS. M. (2018). Journal article reporting standards for quantitative research in psychology: The APA Publications and Communications Board task force report. American Psychologist, 73, 3–25. doi:10.1037/amp0000191
7.
AudigierV.WhiteI. R.JolaniS.DebrayT. P. A.QuartagnoM.CarpenterJ.. . . Resche-RigonM. (2018). Multiple imputation for multilevel data with continuous and binary variables. Statistical Science, 33, 160–183. doi:10.1214/18-STS646
8.
BairdM. D.PaneJ. F. (2019). Translating standardized effects of education programs into more interpretable metrics. Educational Researcher, 48, 217–228. doi:10.3102/0013189X19848729
9.
BorensteinM.HedgesL. V.HigginsJ. P. T.RothsteinH. R. (2009). Introduction to meta-analysis. Chichester, England: Wiley.
10.
Centre for Reviews and Dissemination. (2009). Systematic reviews: CRD’s guidance for undertaking reviews in health care. York, England: University of York.
11.
CheungM. W. L. (2015). metaSEM: An R package for meta-analysis using structural equation modeling. Frontiers in Psychology, 5, 1521.
12.
CitkowiczM.VeveaJ. L. (2017). A parsimonious weight function for modeling publication bias. Psychological Methods, 22(1), 28–41. doi:10.1037/met0000119
13.
CooperH. (2017). Research synthesis and meta-analysis: A step-by-step approach (5th ed.). Thousand Oaks, CA: Sage.
14.
CooperH.HedgesL. V. (2009). Research synthesis as a scientific process. In CooperH.HedgesL. V.ValentineJ. C. (Eds.), The handbook of research synthesis and meta-analysis (2nd ed., pp. 3–18). New York, NY: Russell Sage Foundation.
15.
CooperL. V.HedgesL. V.ValentineJ. C. (Eds.). (2009). The handbook of research synthesis and meta-analysis (2nd ed.). New York, NY: Russell Sage Foundation.
16.
CordrayD. S.MorphyP. (2009). Research synthesis and public policy. In CooperH.HedgesL. V.ValentineJ. C. (Eds.), The handbook of research synthesis and meta-analysis (2nd ed., pp. 473–496). New York, NY: Russell Sage Foundation.
17.
De La RueL.PolaninJ. R.EspelageD. L.PigottT. D. (2017). School-based interventions to reduce dating and sexual violence: A systematic review. Review of Educational Research, 87, 7–34. doi:10.3102/0034654316632061
18.
DickersinK. (1990). The existence of publication bias and risk factors for its occurrence. Journal of the American Medical Association, 263, 1385–1389.
19.
DietrichsonJ.BøgM.FilgesT.JørgensenA.-M. (2017). Academic interventions for elementary and middle school students with low socioeconomic status: A systematic review and meta-analysis. Review of Educational Research, 87, 243–282. doi:10.3102/0034654316687036
20.
DuongM. T.BadalyD.LiuF. F.SchwartzD.McCartyC. A. (2016). Generational differences in academic achievement among immigrant youths: A meta-analytic review. Review of Educational Research, 86, 3–41. doi:10.3102/0034654315577680
21.
EndersC. K.MistlerS. A.KellerB. T. (2016). Multilevel multiple imputation: A review and evaluation of joint modeling and chained equations imputation. Psychological Methods, 21, 222–240. doi:10.1037/met0000063
22.
FergusonC. J.BrannickM. T. (2012). Publication bias in psychological science: Prevalence, methods for identifying and controlling, and implications for the use of meta-analyses. Psychological Methods, 17, 120–128. doi:10.1037/a0024445
GardellaJ. H.FisherB. W.Teurbe-TolonA. R. (2017). A systematic review and meta-analysis of cyber-victimization and educational outcomes for adolescents. Review of Educational Research, 87, 283–308. doi:10.3102/0034654316689136
25.
GlassG. V. (1976). Primary, secondary, and meta-analysis. Educational Researcher, 5(10), 3–8. doi:10.2307/1174772
26.
GoughD.OliverS.ThomasJ. (2017). An introduction to systematic reviews (2nd ed.). London, England: Sage.
27.
GrundS.LüdtkeO.RobitzschA. (2018). Multiple imputation of missing data for multilevel models: Simulations and recommendations. Organizational Research Methods, 21, 111–149. doi:10.1177/1094428117703686
28.
HedgesL. V. (2007). Effect sizes in cluster-randomized designs. Journal of Educational and Behavioral Statistics, 32, 341–370. doi:10.3102/1076998606298043
29.
HedgesL. V. (2011). Effect sizes in three-level cluster-randomized experiments. Journal of Educational and Behavioral Statistics, 36, 346–380. doi:10.3102/1076998610376617
30.
HedgesL. V.PigottT. D. (2004). The power of statistical tests for moderators in meta-analysis. Psychological Methods, 9, 426–445. doi:10.1037/1082-989X.9.4.426
31.
HedgesL. V.TiptonE.JohnsonM. C. (2010). Robust variance estimation in meta-regression with dependent effect size estimates. Research Synthesis Methods, 1(1), 39–65. doi:10.1002/jrsm.5
32.
HigginsJ. P. T.DeeksJ. J. (2011). Selecting studies and collecting data. In HigginsJ. P. T.GreenS. (Eds.), Cochrane handbook for systematic reviews of interventions Version 5.1.0 (pp. 151–187). Hoboken, NJ: Wiley.
33.
HigginsJ. P. T.GreenS. (Eds.). (2011). Cochrane handbook for systematic reviews of interventions (Version 5.1.0). Hoboken, NJ: Wiley.
34.
Institute of Medicine. (2011). Finding what works in health care: Standards for systematic reviews. Washington, DC: National Academies Press. doi:10.17226/13059
35.
IoannidisJ. P. A. (2005). Why most published research findings are false. PLoS Med, 2(8), e124. doi:10.1371/journal.pmed.0020124
36.
JakS.CheungM. W.-L. (2018). Accounting for missing correlation coefficients in fixed-effects MASEM. Multivariate Behavioral Research, 53(1), 1–14. doi:10.1080/00273171.2017.1375886
LipseyM. W.WilsonD. B. (2001). Practical meta-analysis. Thousand Oaks, CA: Sage.
39.
MacaskillP.WalterS. D.IrwigL. (2001). A comparison of methods to detect publication bias in meta-analysis. Statistics in Medicine, 20, 641–654. doi:10.1002/sim.698
MoeyaertM.UgilleM.Natasha BeretvasS.FerronJ.BunuanR.Van den NoortgateW. (2017). Methods for dealing with multiple outcomes in meta-analysis: A comparison between averaging effect sizes, robust variance estimation and multilevel meta-analysis. International Journal of Social Research Methodology, 20, 559–572.
42.
MoherD.LiberatiA.TetzlaffJ.AltmanD. G. (2009). Preferred reporting items for systematic reviews and meta-analyses: The PRISMA statement. PLoS Medicine, 6(7), e1000097. doi:10.1371/journal.pmed.1000097
43.
Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349, aac4716–aac4716. doi:10.1126/science.aac4716
44.
PatallE. A.CooperH.RobinsonJ. C. (2008). Parent involvement in homework: A research synthesis. Review of Educational Research, 78, 1039–1101.
45.
PearsonK. (1904). Report on certain enteric fever inoculation studies. British Medical Journal, 3, 1243–1246.
46.
PigottT. D. (2012). Advances in meta-analysis. New York, NY: Springer.
47.
PigottT. D. (in press). Missing data in meta-analysis. In CooperH.HedgesL. V.ValentineJ. C. (Eds.), The handbook of research synthesis and meta-analysis (3rd ed.). New York, NY: Russell Sage Foundation.
48.
PolaninJ. R. (2018). The consequences of school violence: A systematic review and meta-analysis, Open Science Framework, Retrieved from https://osf.io/6hak7/
49.
PolaninJ. R.HennessyE. A.Tanner-SmithE. E. (2017). A review of meta-analysis packages in R. Journal of Educational and Behavioral Statistics, 42, 206–242. doi:10.3102/1076998616674315
50.
PolaninJ. R.PigottT. D. (2015). The use of meta-analytic statistical significance testing. Research Synthesis Methods, 6, 63–73. doi:10.1002/jrsm.1124
51.
PolaninJ. R.PigottT. D.EspelageD. L.GrotpeterJ. (2019). Best practice guidelines for abstract screening large-evidence systematic reviews and meta-analyses. Research Synthesis Methods, 10, 330–342. doi:10.1002/jrsm.1354
52.
PolaninJ. R.SnilstveitB. (2016). Campbell methods policy note on converting between effect sizes (Version 1.1). Oslo, Norway: Campbell Collaboration. doi:10.4073/cmpn.2016.3
53.
PolaninJ. R.Tanner-SmithE. E.HennessyE. (2016). Estimating the difference between published and unpublished effect sizes: A meta-review. Review of Educational Research, 86, 207–236. doi:10.3102/0034654315582067
54.
PolaninJ. R.TerzianM. (2019). A data-sharing agreement helps to increase researchers’ willingness to share primary data: Results from a randomized controlled trial. Journal of Clinical Epidemiology, 106, 60–69. doi:10.1016/j.jclinepi.2018.10.006
55.
PolaninJ. R.WilliamsR. T. (2016). Overcoming obstacles in obtaining individual participant data for meta-analysis. Research Synthesis Methods, 7, 333–341. doi:10.1002/jrsm.1208
PustejovskyJ. E.RodgersM. A. (2019). Testing for funnel plot asymmetry of standardized mean differences. Research Synthesis Methods, 10, 57–71. doi:10.1002/jrsm.1332
58.
SheaB. J.ReevesB. C.WellsG.ThukuM.HamelC.MoranJ.. . . HenryD. A. (2017). AMSTAR 2: A critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both. British Medical Journal, 358, j4008. doi:10.1136/bmj.j4008
59.
SnilstveitB.VojtkovaM.BhavsarA.StevensonJ.GaarderM. (2016). Evidence & gap maps: A tool for promoting evidence informed policy and strategic research agendas. Journal of Clinical Epidemiology, 79, 120–129.
60.
SterneJ. A.EggerM.MoherD. (2011). Addressing reporting biases. In Cochrane handbook for systematic reviews of interventions (Version 5.1.0). Retrieved from www.handbook.cochrane.org
61.
StewartL.MoherD.ShekelleP. (2012). Why prospective registration of systematic reviews makes sense. Systematic Reviews, 1(1), 7. doi:10.1186/2046-4053-1-7
62.
Tanner-SmithE. E.TiptonE.PolaninJ. R. (2016). Handling complex meta-analytic data structures using robust variance estimates: A tutorial in R. Journal of Developmental and Life-Course Criminology, 2, 85–112. doi:10.1007/s40865-016-0026-5
63.
TiptonE.PustejovskyJ. E.AhmadiH. (2019a). A history of meta-regression: Technical, conceptual, and practical developments between 1974 and 2018. Research Synthesis Methods, 10, 161–179. doi:10.1002/jrsm.1338
64.
TiptonE.PustejovskyJ. E.AhmadiH. (2019b). Current practices in meta-regression in psychology, education, and medicine. Research Synthesis Methods, 10, 180–194. doi:10.1002/jrsm.1339
65.
Van den NoortgateW.López-LópezJ. A.Marín-MartínezF.Sánchez-MecaJ. (2015). Meta-analysis of multiple outcomes: A multilevel approach. Behavior Research Methods, 47, 1274–1294.
66.
VeveaJ. L.HedgesL. V. (1995). A general linear model for estimating effect size in the presence of publication bias. Psychometrika, 60, 419–435. doi:10.1007/BF02294384
67.
ViechtbauerW. (2010). Conducting meta-analyses in R with the metafor package. Journal of Statistical Software, 36, 1–48.
68.
WallaceB. C.SmallK.BrodleyC. E.LauJ.TrikalinosT. A. (2012). Deploying an interactive machine learning system in an evidence-based practice center: abstrackr. In Proceedings of the ACM International Health Informatics Symposium (pp. 819–824). New York, NY: ACM.
WilliamsR. T. (2012). Using robust standard errors to combine multiple regression estimates with meta-analysis (Doctoral dissertation; Order No. 3526372). Available from ProQuest Dissertations & Theses Global. (1081489864)
WoodW.EaglyA. H. (2009). Advantages of certainty and uncertainty. In CooperH.HedgesL. V.ValentineJ. C. (Eds.), The handbook of research synthesis and meta-analysis (2nd ed., pp. 455–472). New York, NY: Russell Sage Foundation.
