Small-Sample Adjustments for Tests of Moderators and Model Fit Using Robust Variance Estimation in Meta-Regression

Abstract

Meta-analyses often include studies that report multiple effect sizes based on a common pool of subjects or that report effect sizes from several samples that were treated with very similar research protocols. The inclusion of such studies introduces dependence among the effect size estimates. When the number of studies is large, robust variance estimation (RVE) provides a method for pooling dependent effects, even when information on the exact dependence structure is not available. When the number of studies is small or moderate, however, test statistics and confidence intervals based on RVE can have inflated Type I error. This article describes and investigates several small-sample adjustments to F-statistics based on RVE. Simulation results demonstrate that one such test, which approximates the test statistic using Hotelling’s T² distribution, is level-α and uniformly more powerful than the others. An empirical application demonstrates how results based on this test compare to the large-sample F-test.

Keywords

meta-analysis cluster robust standard errors

Get full access to this article

View all access options for this article.

References

Ahn

Ames

A. J.

Myers

N. D.

(2012). A review of meta-analyses in education: Methodological strengths and weaknesses. Review of Educational Research, 82, 436–476. doi:10.3102/0034654312458162

Alexander

R. A.

Govern

D. M.

(1994). A new and simpler approximation for ANOVA under variance heterogeneity. Journal of Educational and Behavioral Statistics, 19, 91–101. doi:10.3102/10769986019002091

Bell

R. M.

McCaffrey

D. F.

(2002). Bias reduction in standard errors for linear regression with multi-stage samples. Survey Methodology, 28, 169–181.

Cai

Hayes

A. F.

(2008). A new test of linear hypotheses in OLS regression under heteroscedasticity of unknown form. Journal of Educational and Behavioral Statistics, 33, 21–40. doi:10.3102/1076998607302628

Cooper

H. M.

(2010). Research synthesis and meta-analysis (4th ed.). Thousand Oaks, CA: SAGE.

De Vibe

Bjørndal

Tipton

Hammerstrøm

Kowalski

(2012). Mindfulness based stress reduction (MBSR) for improving health, quality of life, and social functioning in adults. Campbell Systematic Reviews, 8. doi:10.4073/csr.2012.3

Fai

A. H.-T.

Cornelius

(1996). Approximate F-tests of multiple degree of freedom hypotheses in generalized least squares analyses of unbalanced split-plot experiments. Journal of Statistical Computation and Simulation, 54, 363–378.

Fisher

Tipton

(2014). robumeta: An R-package for robust variance estimation in meta-analysis. Retrieved from arXiv preprint arXiv:1503.02220

Gleser

L. J.

Olkin

(2009). Stochastically dependent effect sizes. In Cooper

Hedges

L. V.

Valentine

J. C.

(Eds.), The handbook of research synthesis and meta-analysis (2nd ed., pp. 357–376). New York, NY: Russell Sage Foundation.

10.

Hedberg

E. C.

(2011). ROBUMETA: Stata module to perform robust variance estimation in meta-regression with dependent effect size estimates. Statistical Software Components S457219, Boston College Department of Economics.

11.

Hedges

L. V

Tipton

Johnson

M. C.

(2010). Robust variance estimation in meta-regression with dependent effect size estimates. Research Synthesis Methods, 1, 39–65. doi:10.1002/jrsm.5

12.

Hill

G. W.

(1970). Algorithm 395: Student’s t quantiles. Communications of the ACM, 13, 617–619.

13.

Krishnamoorthy

(2004). Modified Nel and Van der Merwe test for the multivariate Behrens–Fisher problem. Statistics & Probability Letters, 66, 161–169. doi:10.1016/j.spl.2003.10.012

14.

MacKinnon

J. G.

(2013). Thirty years of heteroskedasticity-robust inference. In Chen

Swanson

N. R.

(Eds.), Recent advances and future directions in causality, prediction, and specification analysis (pp. 437–461). New York, NY: Springer New York. doi:10.1007/978-1-4614-1653-1

15.

McCaffrey

D. F.

Bell

R. M.

(2006). Improved hypothesis testing for coefficients in generalized estimating equations with small samples of clusters. Statistics in Medicine, 25, 4081–4098. doi:10.1002/sim.2502

16.

McCaffrey

D. F.

Bell

R. M.

Botts

C. H.

(2001, August). Generalizations of biased reduced linearization. In Proceedings of the Annual Meeting of the American Statistical Association.

17.

Muirhead

R. J.

(1982). Aspects of multivariate statistical theory. New York, NY: John Wiley & Sons.

18.

Nel

van der Merwe

(1986). A solution to the multivariate Behrens-Fisher problem. Communications in Statistics - Theory and Methods, 15, 3719–3735.

19.

Pan

Wall

M. M.

(2002). Small-sample adjustments in using the sandwich variance estimator in generalized estimating equations. Statistics in Medicine, 21, 1429–1441. doi:10.1002/sim.1142

20.

Polanin

J. R.

Pigott

T. D.

(2015). The use of meta-analytic statistical significance testing. Research Synthesis Methods, 6, 63–73. doi:10.1002/jrsm.1124

21.

Raudenbush

S. W.

Bryk

A. S.

(2002). Hierarchical linear models: Applications and data analysis methods (2nd ed.). Thousand Oaks, CA: Sage.

22.

Samson

J. E.

Ojanen

Hollo

(2012). Social goals and youth aggression: Meta-analysis of prosocial and antisocial goals. Social Development, 21, 645–666. doi:10.1111/j.1467-9507.2012.00658.x

23.

Tanner-Smith

E. E.

Tipton

(2014). Robust variance estimation with dependent effect sizes: Practical considerations including a software tutorial in Stata and SPSS. Research Synthesis Methods, 5, 13–30. doi:10.1002/jrsm.1091

24.

Tanner-Smith

E. E.

Wilson

S. J.

Lipsey

M. W.

(2013). The comparative effectiveness of outpatient treatment for adolescent substance abuse: a meta-analysis. Journal of Substance Abuse Treatment, 44, 145–158. doi:10.1016/j.jsat.2012.05.006

25.

Tipton

(2013). Robust variance estimation in meta-regression with binary dependent effects. Research Synthesis Methods, 4, 169–187. doi:10.1002/jrsm.1070

26.

Tipton

(2015). Small sample adjustments for robust variance estimation with meta-regression. Psychological Methods, 20(3), 375–393. doi:10.1037/met0000011

27.

Uttal

D. H.

Meadow

N. G.

Tipton

Hand

L. L.

Alden

A. R.

Warren

Newcombe

N. S.

(2013). The malleability of spatial skills: A meta-analysis of training studies. Psychological Bulletin, 139, 352–402. doi:10.1037/a0028446

28.

Webb

MacKinnon

J. G.

(2013). Wild bootstrap inference for wildly different cluster sizes (No. 1314). Ontario, Canada: Kingston.

29.

Williams

R. T.

(2012). Using robust standard errors to combine multiple regression estimates with meta-analysis. Loyola University. Retrieved from http://ecommons.luc.edu/luc_diss/405/

30.

Wilson

S. J.

Lipsey

M. W.

Tanner-Smith

Huang

C. H.

Steinka-Fry

K. T.

(2011). Dropout prevention and intervention programs: Effects on school completion and dropout Aaong school-aged children and youth: A systematic review. Campbell Systematic Reviews, 8. doi: 10.4073/csr.2011.8

31.

Zhang

J.-T.

(2012). An approximate Hotelling T2-test for heteroscedastic one-way MANOVA. Open Journal of Statistics, 2, 1–11.

32.

Zhang

J.-T.

(2013). Tests of linear hypotheses in the ANOVA under heteroscedasticity. International Journal of Advanced Statistics and Probability, 1, 9–24.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.01 MB

4.79 MB