In this article, the authors compare the multilevel meta-analysis approach with the more traditional meta-analytical approaches. After a description and comparison of the under-lying models and some of the major techniques, the results of the multilevel approach are compared with those of the traditional approaches, using a simulation study. The results of the simulation study suggest that the maximum likelihood multilevel approach is in general superior to the fixed-effects approaches, unless only a small number of studies is available. For models without moderators, the results of the multilevel approach, however, are not substantially different from the results of the traditional random-effects approaches.
Get full access to this article
View all access options for this article.
References
1.
Bosker, R. J., & Snijders, T. A. B. (1990). Statistische aspecten van multi-niveau onderzoek. Tijdschrift Voor Onderwijsresearch, 15, 317-329.
2.
Bryk, A. S., & Raudenbush, S. W. (1992). Hierarchical linear models: Applications and data analysis methods. Newbury Park, CA: Sage.
3.
Capraro, M. M., Capraro, R. M., & Henson, R. K. (2001). Measurement error of scores on the Mathematics Anxiety Rating scale across studies. Educational and Psychological Measurement, 61, 373-386.
4.
Caruso, J. C. (2000). Reliability generalization of the NEO personality scales. Educational and Psychological Measurement, 60, 236-254.
5.
Caruso, J. C., Witkiewitz, K., Belcourt-Dittloff, A., & Gottlieb, J. (2001). Reliability of scores from the Eysenck Personality Questionnaire: A reliability generalization (RG) study. Educational and Psychological Measurement, 61, 675-682.
6.
Cochran, W. G. (1954). The combination of estimates from different experiments. Biometrics, 10, 101-129.
7.
Cornwell, J. M., & Ladd, R. T. (1993). Power and accuracy of the Schmidt and Hunter meta-analytic procedures. Educational and Psychological Measurement, 53, 877-895.
8.
Cronbach, L. J. (1980). Toward reform of program evaluation. San Francisco: Jossey-Bass.
9.
DerSimonian, R., & Laird, N. (1986). Meta-analysis in clinical trials. Controlled Clinical Trials, 7, 177-188.
10.
Draper, D. (1995). Inference and hierarchical modeling in the social sciences. Journal of Educational and Behavioral Statistics, 20(2), 115-147.
11.
Erez, A., Bloom, M. C., & Wells, M. T. (1996). Using random rather than fixed effects models in meta-analysis: Implications for situational specificity and validity generalization. Personnel Psychology, 49, 275-306.
12.
Glass, G. V (1976). Primary, secondary, and meta-analysis of research. Educational Researcher, 5, 3-8.
13.
Glass, G. V (1977). Integrating findings: The meta-analysis of research. Review of Research in Education, 5, 351-379.
14.
Glass, G. V, MacGaw, B., & Smith, M. L. (1981). Meta-analysis in social research. Beverly Hills, CA: Sage.
15.
Goldstein, H. (1995). Multilevel statistical models. London: Edward Arnold.
16.
Goldstein, H., Rasbash, J., Plewis, I., Draper, D., Browne, W., Yang, M., et al. (1998). A user's guide to MLwiN. London: Multilevel Models Project, University of London.
17.
Goldstein, H., & Yang, M. (2000). Meta-analysis using multilevel models with an application to the study of class size effects. Journal of the Royal Statistical Society, Series C: Applied Statistics, 49, 339-412.
18.
Harwell, M. (1997). An empirical study of Hedges's homogeneity test. Psychological Methods, 2, 219-231.
19.
Hedges, L. V. (1981). Distribution theory for Glass's estimator of effect size and related estimators. Journal of Educational Statistics, 6, 107-128.
20.
Hedges, L. V. (1983). A random-effects model for effect sizes. Psychological Bulletin, 93, 388-395.
21.
Hedges, L. V. (1994). Fixed effects models. In H. Cooper & L. V. Hedges (Eds.), The handbook of research synthesis (pp. 285-300). New York: Russell Sage Foundation.
22.
Hedges, L. V., & Olkin, I. (1985). Statistical methods for meta-analysis. Orlando, FL: Academic Press.
23.
Hedges, L. V., & Vevea, J. L. (1998). Fixed- and random-effects models in meta-analysis. Psychological Methods, 3, 486-504.
24.
Helms, J. E. (1999). Another meta-analysis of the White Racial Identity Attitude scale's Cronbach alphas: Implications for validity. Measurement and Evaluation in Counseling and Development, 32, 122-137.
25.
Henson, R. K., Kogan, L. R., & Vacha-Haase, T. (2001). A reliability generalization study of the Teacher Efficacy scale and related instruments. Educational and Psychological Measurement, 61, 404-420.
26.
Hox, J. J., & de Leeuw, E. D. (1997). Meta-analyse via multiniveau-modellen. Tijdschrift Voor Onderwijsresearch, 22, 16-28.
27.
Hunter, J. E., & Schmidt, F. L. (1990). Methods of meta-analysis: Correcting error and bias in research findings. Newbury Park, CA: Sage.
28.
Hunter, J. E., Schmidt, F. L., & Jackson, G. B. (1982). Meta-analysis: Cumulating findings across research. Beverly Hills, CA: Sage.
29.
Johnson, B. T., Mullen, B., & Salas, E. (1995). Comparison of three major meta-analytic approaches. Journal of Applied Psychology, 80, 94-106.
30.
Kreft, I. G. G., & de Leeuw, J. (1998). Introducing multilevel modeling. London: Sage.
31.
Meijer, E., van der Leeden, R., & Busing, F. M. T. A. (1995). Implementing the bootstrap for multilevel models. Multilevel Modelling Newsletter, 7(2), 7-11.
32.
Mengersen, K. L., Tweedie, R. L., & Biggerstaff, B. J. (1995). The impact of method choice in meta-analysis. Australian Journal of Statistics, 37, 19-44.
33.
Mok, M. (1995). Sample size requirements for 2-level designs in educational research. Multilevel Modelling Newsletter, 7(2), 11-15.
34.
Mosteller, F., & Colditz, G. A. (1996). Understanding research synthesis (meta-analysis). Annual Review Public Health, 17, 1-23.
35.
National Research Council. (1992). Combining information: Statistical issues and opportunities for research. Washington, DC: National Academy Press.
36.
Olkin, I. (1990). History and goals. In K. W. Wachter & M. L. Straf (Eds.), The future of meta-analysis (pp. 3-10). New York: Russell Sage Foundation.
37.
Overton, R. C. (1998). A comparison of fixed-effects and mixed (random effects) models for meta-analysis tests of moderator variable effects. Psychological Methods, 3, 354-379.
38.
Poole, C., & Greenland, S. (1999). Random-effects meta-analyses are not always conservative. American Journal of Epidemiology, 150, 469-475.
39.
Raudenbush, S. W., & Bryk, A. S. (1985). Empirical Bayes meta-analysis. Journal of Educational Statistics, 10, 75-98.
40.
Rosenthal, R. (1961, August). On the social psychology of the psychological experiment: With particular reference to experimenter bias. Paper presented at the meeting of the American Psychological Association, New York.
41.
Rosenthal, R. (1991). Meta-analytic procedures for social research. Newbury Park, CA: Sage.
42.
Rosenthal, R., & Rubin, D. B. (1978). Interpersonal expectancy effects: The first 345 studies. Behavioral and Brain Sciences, 3, 377-415.
43.
Rosenthal, R., & Rubin, D. B. (1982). Comparing effect sizes of independent studies. Psychological Bulletin, 92, 500-504.
44.
Rubin, D. B. (1981). Estimation in parallel randomized experiments. Journal of Educational Statistics, 6, 377-400.
45.
Sagie, A., & Koslowsky, M. (1993). Detecting moderators with meta-analysis: an evaluation and comparison of techniques. Personnel Psychology, 46, 629-640.
46.
Sanchez-Meca, J., & Marin-Martinez, F. (1998). Testing continuous moderators in meta-analysis: A comparison of procedures. British Journal of Mathematical and Statistical Psychology, 51, 311-326.
47.
Schmidt, F. L., & Hunter, J. E. (1977). Development of a general solution to the problem of validity generalization. Journal of Applied Psychology, 62, 529-540.
48.
Snijders, T. A. B., & Bosker, R. J. (1999). Multilevel analysis: An introduction to basic and advanced multilevel modeling. London: Sage.
49.
Takkouche, B., Cadarso-Suarez, C., & Spiegelman, D. (1999). Evaluation of old and new tests of heterogeneity in epidemiologic meta-analysis. American Journal of Epidemiology, 150, 206-215.
50.
Thompson, S. G., & Pocock, S. J. (1991). Can meta-analyses be trusted? Lancet, 338, 1127-1130.
Vacha-Haase, T. (1998). Reliability generalization: Exploring variance in measurement error affecting score reliability across studies. Educational and Psychological Measurement, 58, 6-20.
53.
Vacha-Haase, T., Kogan, L., Tani, C. R., & Woodall, R. A. (2001). Reliability generalization: Exploring reliability coefficients of MMPI clinical scales scores. Educational and Psychological Measurement, 61, 45-59.
54.
Van den Noortgate, W., & Onghena, P. (2001). Estimating the standardized mean difference in a meta-analysis: Bias, precision and mean squared error. Manuscript submitted for publication.
55.
Viswesvaran, C., & Ones, D. (2000). Measurement error in “Big Five Factors” personality assessment: Reliability generalization across studies and measures. Educational and Psychological Measurement, 60, 224-235.
56.
Viswesvaran, C., & Sanchez, J. I. (1998). Moderator search in meta-analysis: A review and cautionary note on existing approaches. Educational and Psychological Measurement, 58, 77-87.
57.
Yin, P., & Fan, X. (2000). Assessing the reliability of Beck Depression Inventory scores: Reliability generalization across studies. Educational and Psychological Measurement, 60, 201-223.
58.
Zeger, S., & Karim, M. (1991). Generalized linear models with random effects: A Gibbs sampling approach. Journal of the American Statistical Association, 86, 79-86.
