Intervention studies represent webs of interrelated substantive and methodological characteristics that take on different patterns in different studies and different intervention areas. All too often, meta-analysts do not give close attention to the possibility that these interrelated differences among studies are related in complex ways to study effect sizes and, consequently, run considerable risk of reporting results that are misleading or flatly wrong. To remedy this situation, improvements are needed in both the method and practice of meta-analysis so that greater attention can be given to effect size variation, the generalizability of study results, and the systematic multivariate relationships between study characteristics and the effect sizes reported in those studies.
Get full access to this article
View all access options for this article.
References
1.
Becker, B. J.
(1996). The generalizability of empirical research results. In C. P. Benbow & D. J. Lubinski (Eds.), Intellectual talent: Psychometric and social issues (pp. 362–383). Baltimore: Johns Hopkins University Press.
2.
Biggerstaff, B. J.
, & Tweedie, R. L. (1997). Incorporating variability in estimates of heterogeneity in the random effects model in meta-analysis. Statistics in Medicine, 16, 753–768.
3.
Cook, T. D.
(1993). A theory of the generalization of causal relationships. New Directions for Program Evaluation, 57, 39–82.
4.
DerSimonian, R.
, & Laird, N. (1986). Meta-analysis in clinical trials. Controlled Clinical Trials, 7, 177–188.
5.
DerSimonian, R.
, & Levine, R. J. (1999). Resolving discrepancies between a meta-analysis and a subsequent large controlled trial. Journal of the American Medical Association, 282(7), 664–670.
6.
Harwell, M.
(1997). An empirical study of Hedge’s homogeneity test. Psychological Methods, 2(2), 219–231.
7.
Hedges, L. V.
(1982). Estimating effect size from a series of independent experiments. Psychological Bulletin, 92, 490–499.
8.
Hedges, L. V.
, & Olkin, I. (1985). Statistical methods for meta-analysis. San Diego, CA: Academic Press.
9.
Heinsman, D. T.
, & Shadish, W. R. (1996). Assignment methods in experimentation: When do nonrandomized experiments approximate answers from randomized experiments?Psychological Methods, 1(2), 154–169.
10.
Hunter, J. E.
, & Schmidt, F. L. (1990). Methods of meta-analysis: Correcting error and bias in research findings. Newbury Park, CA: Sage.
11.
Ioannidis, J.P.A.
, Cappelleri, J. C., & Lau, J. (1998). Issues in comparisons between meta-analyses and large trials. Journal of the American Medical Association, 279(14), 1089–1093.
12.
Juni, P.
, Witschi, A., Bloch, R., & Egger, M. (1999). The hazards of scoring the quality of clinical trials for meta-analysis. Journal of the American Medical Association, 282(11), 1054–1060.
13.
LeLorier, J.
, Gregoire, G., Benhaddad, A., Lapierre, J., & Derderian, F. (1997). Discrepancies between meta-analyses and subsequent large randomized, controlled trials. New England Journal of Medicine, 337, 536–542.
14.
Lipsey, M. W.
(1992). Juvenile delinquency treatment: A meta-analytic inquiry into the variability of effects. In T. D. Cook, H. Cooper, D. S. Cordray, H. Hartmann, L. V. Hedges, R. J. Light, T. A. Louis, & F. Mosteller (Eds.), Meta-analysis for explanation: A casebook. New York: Russell Sage.
15.
Lipsey, M. W.
(1999). Can rehabilitative programs reduce the recidivism of juvenile offenders? An inquiry into the effectiveness of practical programs. Virginia Journal of Social Policy and the Law, 6(3), 101–129.
16.
Lipsey, M. W.
, & Wilson, D. B. (1993). The efficacy of psychological, educational, and behavioral treatment: Confirmation from meta-analysis. American Psychologist, 48(12), 1181–1209.
17.
Matt, G. E.
, & Navarro, A. M. (1997). What meta-analyses have and have not taught us about psychotherapy effects: A review and future directions. Clinical Psychology Review, 17(1), 1–32.
18.
Mosteller, F.
, & Colditz, G. A. (1996). Understanding research synthesis (meta-analysis). Annual Review of Public Health, 17, 1–23.
19.
Roberts, G.
(1999). Age effects and health appraisal: A meta-analysis. Journals of Gerontology: Series B: Psychological Sciences and Social Sciences, 54B(1), S24-S30.
20.
Rooney, B. L.
, & Murray, D. M. (1996). A meta-analysis of smoking prevention programs after adjustment for errors in the unit of analysis. Health Education Quarterly, 23(1), 48–64.
21.
Rosenthal, R.
, & Rubin, D. B. (1982). Comparing effect sizes of independent studies. Psychological Bulletin, 92, 500–504.
22.
Rubin, D. B.
(1992). Meta-analysis: Literature synthesis or effect-size surface estimation?Journal of Educational Statistics, 17, 363–374.
23.
Sanchez-Meca, J.
, & Marin-Martinez, F. (1997). Homogeneity tests in meta-analysis: A Monte Carlo comparison of statistical power and Type I error. Quality and Quantity, 31(4), 385–399.
24.
Schulz, K. F.
, Chalmers, I., Hayes, R. J., & Altman, D. G. (1995). Empirical evidence of bias: Dimensions of methodological quality associated with estimates of treatment effects in controlled trials. Journal of the American Medical Association, 273, 408–412.
25.
Shadish, W. R.
, & Ragsdale, K. (1996). Random versus nonrandom assignment in controlled experiments: Do you get the same answer?Journal of Consulting and Clinical Psychology, 64(6), 1290–1305.
26.
Sharpe, D.
(1997). Of apples and oranges, file drawers and garbage: Why validity issues in meta-analysis will not go away. Clinical Psychology Review, 17, 881–901.
27.
Smith, M. L.
, Glass, G. V., & Miller, T. I. (1980). The benefits of psychotherapy. Baltimore: John Hopkins University Press.
28.
Stewart, L. A.
, Parmar, M. K., & Tierney, J. F. (1998). Meta-analyses and large randomized, controlled trials [Letter]. New England Journal of Medicine, 338, 61-61.
29.
Weiss, B.
, & Weisz, J. R. (1990). The impact of methodological factors on child psychotherapy outcome research: A meta-analysis for researchers. Journal of Abnormal Child Psychology, 18, 639–670.
30.
Wilson, D. B. (1995). The role of method in treatment effect estimates: Evidence from psychological, behavioral, and educational treatment intervention meta-analyses. Unpublished doctoral dissertation, Claremont Graduate School, Claremont, CA.
