Meta-analytic models for dependent effect sizes have grown increasingly sophisticated over the last few decades, which has created challenges for a priori power calculations. We introduce power approximations for tests of average effect sizes based upon several common approaches for handling dependent effect sizes. In a Monte Carlo simulation, we show that the new power formulas can accurately approximate the true power of meta-analytic models for dependent effect sizes. Lastly, we investigate the Type I error rate and power for several common models, finding that tests using robust variance estimation provide better Type I error calibration than tests with model-based variance estimation. We consider implications for practice with respect to selecting a working model and an inferential approach.
AhnS.AmesA. J.MyersN. D. (2012). A review of meta-analyses in education: Methodological strengths and weaknesses. Review of Educational Research, 82(4), 436–476.
2.
DietrichsonJ.BøgM.FilgesT.Klint JørgensenA-M. (2017). Academic interventions for elementary and middle school students with low socioeconomic status: A systematic review and meta-analysis. Review of Educational Research, 87(2), 243–282.
3.
Fernández-CastillaB.AloeA. M.DeclercqL.JamshidiL.BeretvasS. N.OnghenaP.Van den NoortgateW. (2020). Estimating outcome-specific effects in meta-analyses of multiple outcomes: A simulation study. Behavior Research Methods, 53(2), 702–717. https://doi.org/10.3758/s13428-020-01459-4
4.
FisherZ.TiptonE. (2015). robumeta: An R-package for robust variance estimation in meta-analysis. ArXiv Preprint. ArXiv:1503.02220.
5.
GiesbrechtF. G.BurnsJ. C. (1985). Two-stage analysis based on a mixed model: Large-sample asymptotic theory and small-sample simulation results. Biometrics, 41(2), 477–486. https://doi.org/10.2307/2530872
6.
HartungJ.KnappG. (2001). On tests of the overall treatment effect in meta-analysis with normally distributed responses. Statistics in Medicine, 20(12), 1771–1782. https://doi.org/10.1002/sim.791
7.
HedgesL. V. (2007). Effect sizes in cluster-randomized designs. Journal of Educational and Behavioral Statistics, 32(4), 341–370. https://doi.org/10.3102/1076998606298043
8.
HedgesL. V.OlkinI. (1985). Statistical methods for meta-analysis. Academic Press.
9.
HedgesL. V.PigottT. D. (2001). The power of statistical tests in meta-analysis. Psychological Methods, 6(3), 203.
10.
HedgesL. V.PigottT. D. (2004). The power of statistical tests for moderators in meta-analysis. Psychological Methods, 9(4), 426.
11.
HedgesL. V.TiptonE.JohnsonM. C. (2010). Robust variance estimation in meta-regression with dependent effect size estimates. Research Synthesis Methods, 1(1), 39–65. https://doi.org/10.1002/jrsm.5
12.
JacksonD.TurnerR. (2017). Power analysis for random-effects meta-analysis. Research Synthesis Methods, 8(3), 290–302.
13.
KenwardM. G.RogerJ. H. (2009). An improved approximation to the precision of fixed effects from restricted maximum likelihood. Computational Statistics & Data Analysis, 53(7), 2583–2595.
LindenA. H.HönekoppJ. (2021). Heterogeneity of research results: A new perspective from which to assess and promote progress in psychological science. Perspectives on Psychological Science, 16(2), 358–376.
17.
LukeS. G. (2017). Evaluating significance in linear mixed-effects models in R. Behavior Research Methods, 49(4), 1494–1502. https://doi.org/10.3758/s13428-016-0809-y
18.
MoeyaertM.UgilleM.Natasha BeretvasS.FerronJ.BunuanR.Van den NoortgateW. (2017). Methods for dealing with multiple outcomes in meta-analysis: A comparison between averaging effect sizes, robust variance estimation and multilevel meta-analysis. International Journal of Social Research Methodology, 20(6), 559–572. https://doi.org/10.1080/13645579.2016.1252189
19.
Olsson-CollentineA.WichertsJ. M.van AssenM. A. L. M. (2020). Heterogeneity in direct replications in psychology and its association with effect size. Psychological Bulletin, 146(10), 922.
20.
PigottT. (2012). Advances in meta-analysis. Springer Science & Business Media.
21.
PolaninJ. R.HennessyE. A.TsujiS. (2020). Transparency and reproducibility of meta-analyses in psychology: A meta-review. Perspectives on Psychological Science, 15(4), 1026–1041.
22.
PustejovskyJ. E. (2020). ClubSandwich: Cluster-robust (sandwich) variance estimators with small-sample corrections. R package version 0.5.0 (0.5.2). cran.r-project.org.
23.
PustejovskyJ. E.TiptonE. (2021). Meta-analysis with robust variance estimation: Expanding the range of working models. Prevention Science, 23(1), 425–438. https://doi.org/10.1007/s11121-021-01246-3
TiptonE. (2015). Small sample adjustments for robust variance estimation with meta-regression. Psychological Methods, 20(3), 375–393. https://doi.org/10.1037/met0000011
27.
TiptonE.PustejovskyJ. E. (2015). Small-sample adjustments for tests of moderators and model fit using robust variance estimation in meta-regression. Journal of Educational and Behavioral Statistics, 40(6), 604–634. https://doi.org/10.3102/1076998615606099
28.
TiptonE.PustejovskyJ. E.AhmadiH. (2019). Current practices in meta-regression in psychology, education, and medicine. Research Synthesis Methods, 10(2), 180–194. https://doi.org/10.1002/jrsm.1339
29.
ValentineJ. C.PigottT. D.RothsteinH. R. (2010). How many studies do you need?: A primer on statistical power for meta-analysis. Journal of Educational and Behavioral Statistics, 35(2), 215–247. https://doi.org/10.3102/1076998609346961
30.
Van den NoortgateW.López-LópezJ.Marín-MartínezF.Sánchez-MecaJ. (2013). Three-level meta-analysis of dependent effect sizes. Behavior Research Methods, 45(2), 576–594. https://doi.org/10.3758/s13428-012-0261-6
31.
Van den NoortgateW.López-LópezJ. A.Marín-MartínezF.Sánchez-MecaJ. (2014). Meta-analysis of multiple outcomes: A multilevel approach. Behavior Research Methods, 47(4), 1274–1294. https://doi.org/10.3758/s13428-014-0527-2
32.
VenablesW. N.RipleyB. D. (2002). Modern applied statistics with S, Fourth edition. Springer.
33.
ViechtbauerW. (2010). Conducting meta-analyses in R with the metafor package. Journal of Statistical Software, 36(3), 1–48. https://doi.org/10.18637/jss.v036.i03
34.
WhiteH. (1982). Maximum likelihood estimation of misspecified models. Econometrica, 50(1), 1. https://doi.org/10.2307/1912526
35.
ZumboB. D.HubleyA. M. (1998). A note on misconceptions concerning prospective and retrospective power. Journal of the Royal Statistical Society: Series D (The Statistician), 47(2), 385–388. https://doi.org/10.1111/1467-9884.00139
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
