Meta-analyses of educational research findings frequently involve statistically dependent effect size estimates. Meta-analysts have often addressed dependence issues using ad hoc approaches that involve modifying the data to conform to the assumptions of models for independent effect size estimates, such as by aggregating estimates to obtain one summary estimate per study, conducting separate analyses of distinct subgroups of estimates, or combinations thereof. We show that these ad hoc approaches correspond exactly to certain multivariate models for dependent effect sizes. Specifically, we describe classes of multivariate random effects models that have likelihoods equivalent to those of models for effect sizes that have been averaged by study, classified into subgroups, or both. The equivalencies also apply to robust variance estimation methods.
AhnS.AmesA. J.MyersN. D. (2012). A review of meta-analyses in education: Methodological strengths and weaknesses. Review of Educational Research, 82(4), 436–476. https://doi.org/10.3102/0034654312458162
2.
BeckerB. J. (1992). Using results from replicated studies to estimate linear models. Journal of Educational Statistics, 17(4), 341–362. https://doi.org/10.3102/10769986017004341
3.
BeckerB. J. (2000). Multivariate meta-analysis. In BrownS. D.TinsleyH. E. A. (Eds.), Handbook of applied multivariate statistics and mathematical modeling (pp. 499–525). Academic Press. https://doi.org/10.1016/B978-012691360-6/50018-5
ChenY.HongC.RileyR. D. (2014). An alternative pseudolikelihood method for multivariate random-effects meta-analysis. Statistics in Medicine, 34(3), 361–380. https://doi.org/10.1002/sim.6350
7.
CheungM. W.-L. (2008). A model for integrating fixed-, random-, and mixed-effects meta-analyses into structural equation modeling. Psychological Methods, 13(3), 182–202. https://doi.org/10.1037/a0013163
8.
CheungM. W.-L. (2013). Multivariate meta-analysis as structural equation models. Structural Equation Modeling: A Multidisciplinary Journal, 20(3), 429–454. https://doi.org/10.1080/10705511.2013.797827
9.
CheungM. W.-L. (2014). Modeling dependent effect sizes with three-level meta-analyses: A structural equation modeling approach. Psychological Methods, 19(2), 211–229. https://doi.org/10.1037/a0032968
10.
CheungM. W.-L. (2019). A guide to conducting a meta-analysis with non-independent effect sizes. Neuropsychology Review, 29(4), 387–396. https://doi.org/10.1007/s11065-019-09415-6
11.
Clinton-LisellV. (2021). Stop multitasking and just read: Meta-analyses of multitasking’s effects on reading performance and reading time. Journal of Research in Reading, 44(4), 787–816. https://doi.org/10.1111/1467-9817.12372
12.
CooperH. M. (1998). Synthesizing research: A guide for literature reviews (3rd ed.). Sage.
13.
CooperH. M. (2016). Research synthesis and meta-analysis (5th ed.). Sage.
14.
De WitL.MarsiskeM.O’SheaD.KesselsR. P.KuraszA. M.DeFeisB.SchaeferN.SmithG. E. (2021). Procedural learning in individuals with amnestic mild cognitive impairment and Alzheimer’s dementia: A systematic review and meta-analysis. Neuropsychology Review, 31, 103–114. https://doi.org/10.1007/s11065-020-09449-1
EmslanderV.SchererR. (2022). The relation between executive functions and math intelligence in preschool children: A systematic review and meta-analysis. Psychological Bulletin, 148(5–6), 337. https://doi.org/10.1037/bul0000369
17.
Fernández-CastillaB.AloeA. M.DeclercqL.JamshidiL.BeretvasS. N.OnghenaP.Van den NoortgateW. (2020). Estimating outcome-specific effects in meta-analyses of multiple outcomes: A simulation study. Behavior Research Methods, 53, 702–717. https://doi.org/10.3758/s13428-020-01459-4
18.
Fernández-CastillaB.MaesM.DeclercqL.JamshidiL.BeretvasS. N.OnghenaP.Van den NoortgateW. (2019). A demonstration and evaluation of the use of cross-classified random-effects models for meta-analysis. Behavior Research Methods, 51(3), 1286–1304. https://doi.org/10.3758/s13428-018-1063-2
FurenesM. I.KucirkovaN.BusA. G. (2021). A comparison of children’s reading on paper versus screen: A meta-analysis. Review of Educational Research, 91(4), 483–517. https://doi.org/10.3102/0034654321998074
21.
GaskellC.Simmonds-BuckleyM.KellettS.StocktonC.SomervilleE.RogersonE.DelgadilloJ. (2023). The effectiveness of psychological interventions delivered in routine practice: Systematic review and meta-analysis. Administration and Policy in Mental Health and Mental Health Services Research, 50(1), 43–57. https://doi.org/10.1007/s10488-022-01225-y
22.
GleserL. J.OlkinI. (2009). Stochastically dependent effect sizes. In CooperH.HedgesL. V.ValentineJ. C. (Eds.), The handbook of research synthesis and meta-analysis (2nd ed., pp. 357–376). Russell Sage Foundation.
23.
GrahamS.HebertM.FishmanE.RayA. B.RouseA. G. (2020). Do children classified with specific language impairment have a learning disability in writing? A meta-analysis. Journal of Learning Disabilities, 53(4), 292–310. https://doi.org/10.1177/0022219420917338
24.
HadfieldJ. D.NakagawaS. (2010). General quantitative genetic methods for comparative biology: Phylogenies, taxonomies and multi-trait models for continuous and categorical characters. Journal of Evolutionary Biology, 23(3), 494–508. https://doi.org/10.1111/j.1420-9101.2009.01915.x
25.
HartungJ.KnappG. (2001). On tests of the overall treatment effect in meta-analysis with normally distributed responses. Statistics in Medicine, 20(12), 1771–1782. https://doi.org/10.1002/sim.791
26.
HedbergE. C. (2014). ROBUMETA: Stata module to perform robust variance estimation in meta-regression with dependent effect size estimates. Statistical Software Components S457219. Boston College Department of Economics.
27.
HedgesL. V.OlkinI. (1985). Statistical methods for meta-analysis. Academic Press.
28.
HedgesL. V.TiptonE.JohnsonM. C. (2010). Robust variance estimation in meta-regression with dependent effect size estimates. Research Synthesis Methods, 1(1), 39–65. https://doi.org/10.1002/jrsm.5
29.
HongC.RileyR. D.ChenY. (2018). An improved method for bivariate meta-analysis when within-study correlations are unknown. Research Synthesis Methods, 9(1), 73–88. https://doi.org/10.1002/jrsm.1274
30.
JacksonD. (2013). Confidence intervals for the between-study variance in random effects meta-analysis using generalised Cochran heterogeneity statistics. Research Synthesis Methods, 4(3), 220–229. https://doi.org/10.1002/jrsm.1081
31.
JacksonD.RileyR. D.WhiteI. R. (2011). Multivariate meta-analysis: Potential and promise. Statistics in Medicine, 30(20), 2481–2498. https://doi.org/10.1002/sim.4172
32.
JacksonD.WhiteI. R.RileyR. D. (2013). A matrix-based method of moments for fitting the multivariate random effects model for meta-analysis and meta-regression. Biometrical Journal, 55(2), 231–245. https://doi.org/10.1002/bimj.201200152
33.
JoshiM.PustejovskyJ. E.BeretvasS. N. (2022). Cluster wild bootstrapping to handle dependent effect sizes in meta-analysis with a small number of studies. Research Synthesis Methods, 13(4), 457–477. https://doi.org/10.1002/jrsm.1554
34.
JungH.SeoE.HanE.HendersonM. D.PatallE. A. (2020). Prosocial modeling: A meta-analytic review and synthesis. Psychological Bulletin, 146(8), 635–663. https://doi.org/10.1037/bul0000235
35.
KalaianH. A.RaudenbushS. W. (1996). A multivariate mixed linear model for meta-analysis. Psychological Methods, 1(3), 227–235. https://doi.org/10.1037/1082-989X.1.3.227
36.
KoenkaA. C.Linnenbrink-GarciaL.MoshontzH.AtkinsonK. M.SanchezC. E.CooperH. (2021). A meta-analysis on the impact of grades and comments on academic motivation and achievement: A case for written feedback. Educational Psychology, 41(7), 922–947. https://doi.org/10.1080/01443410.2019.1659939
37.
KonstantopoulosS. (2011). Fixed effects and variance components estimation in three-level meta-analysis. Research Synthesis Methods, 2(1), 61–76. https://doi.org/10.1002/jrsm.35
38.
LajeunesseM. J. (2009). Meta-analysis and the comparative phylogenetic method. The American Naturalist, 174(3), 369–381. https://doi.org/10.1086/603628
39.
LeeJ.BeretvasS. N. (2023). Comparing methods for handling missing covariates in meta-regression. Research Synthesis Methods, 14(1), 117–136. https://doi.org/10.1002/jrsm.1585
40.
López-LópezJ. A.PageM. J.LipseyM. W.HigginsJ. P. T. (2018). Dealing with effect size multiplicity in systematic reviews and meta-analyses. Research Synthesis Methods, 9(3), 336–351. https://doi.org/10.1002/jrsm.1310
41.
McCaffreyD. F.BellR. M.BottsC. H. (2001). Generalizations of biased reduced linearization. Proceedings of the Annual Meeting of the American Statistical Association, August 5–9, 2001.
42.
NakagawaS.SantosE. S. A. (2012). Methodological issues and advances in biological meta-analysis. Evolutionary Ecology, 26(5), 1253–1274. https://doi.org/10.1007/s10682-012-9555-5
43.
PageM. J.McKenzieJ. E.ChauM.GreenS. E.ForbesA. (2015). Methods to select results to include in meta-analyses deserve more consideration in systematic reviews. Journal of Clinical Epidemiology, 68(11), 1282–1291. https://doi.org/10.1016/j.jclinepi.2015.02.009
44.
PustejovskyJ. E.TiptonE. (2022). Meta-analysis with robust variance estimation: Expanding the range of working models. Prevention Science, 23(3), 425–438. https://doi.org/10.1007/s11121-021-01246-3
RobertsB. R.MacLeodC. M.FernandesM. A. (2022). The enactment effect: A systematic review and meta-analysis of behavioral, neuroimaging, and patient studies. Psychological Bulletin, 148(5–6), 397–434. https://doi.org/10.1037/bul0000360
47.
RosenthalR.RubinD. B. (1986). Meta-analytic procedures for combining studies with multiple effect sizes. Psychological Bulletin, 99(3), 400–406. https://doi.org/10.1037/0033-2909.99.3.400
48.
SchauerJ. M.LeeJ.DiazK.PigottT. D. (2022). On the bias of complete- and shifting-case meta-regressions with missing covariates. Research Synthesis Methods, 13(4), 489–507. https://doi.org/10.1002/jrsm.1558
49.
SearleS. R.CasellaG.McCullochC. E. (2009). Variance components. John Wiley.
50.
SeraF.ArmstrongB.BlangiardoM.GasparriniA. (2019). An extended mixed-effects framework for meta-analysis. Statistics in Medicine, 38(29), 5429–5444. https://doi.org/10.1002/sim.8362
51.
TiptonE. (2015). Small sample adjustments for robust variance estimation with meta-regression. Psychological Methods, 20(3), 375–393. https://doi.org/10.1037/met0000011
52.
TiptonE.PustejovskyJ. E. (2015). Small-sample adjustments for tests of moderators and model fit using robust variance estimation in meta-regression. Journal of Educational and Behavioral Statistics, 40(6), 604–634. https://doi.org/10.3102/1076998615606099
53.
TiptonE.PustejovskyJ. E.AhmadiH. (2019). Current practices in meta-regression in psychology, education, and medicine. Research Synthesis Methods, 10(2), 180–194. https://doi.org/10.1002/jrsm.1339
54.
Van den NoortgateW.López-LópezJ. A.Marn-MartnezF.Sánchez-MecaJ. (2013). Three-level meta-analysis of dependent effect sizes. Behavior Research Methods, 45(2), 576–594. https://doi.org/10.3758/s13428-012-0261-6
55.
Van den NoortgateW.López-LópezJ. A.Marn-MartnezF.Sánchez-MecaJ. (2015). Meta-analysis of multiple outcomes: A multilevel approach. Behavior Research Methods, 47(4), 1274–1294. https://doi.org/10.3758/s13428-014-0527-2
56.
van HouwelingenH. C.ArendsL. R.StijnenT. (2002). Advanced methods in meta-analysis: Multivariate approach and meta-regression. Statistics in Medicine, 21(4), 589–624. https://doi.org/10.1002/sim.1040
ViechtbauerW. (2010). Conducting meta-analyses in R with the metafor package. Journal of Statistical Software, 36(3), 1–48. https://doi.org/10.18637/jss.v036.i03
60.
ViechtbauerW.López-LópezJ. A. (2022). Location-scale models for meta-analysis. Research Synthesis Methods, 13(6), 697–715. https://doi.org/10.1002/jrsm.1562
61.
WeiY.HigginsJ. P. (2013). Estimating within-study covariances in multivariate meta-analysis with multiple outcomes. Statistics in Medicine, 32(7), 1191–1205. https://doi.org/10.1002/sim.5679
62.
WhiteI. R.BarrettJ. K.JacksonD.HigginsJ. P. T. (2012). Consistency and inconsistency in network meta-analysis: Model estimation using multivariate meta-regression. Research Synthesis Methods, 3(2), 111–125. https://doi.org/10.1002/jrsm.1045
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
