Whether you are a policy maker or social scientist, you are slowly being drowned in a sea of often inconsistent research data. Proponents of meta-analysis claim that such data can be objectively and usefully summarized for you. The author notes how the assumptions of the meta-analytic model preclude the synthesis of experimental data (which has a clear cause-and-effect logic) with quasi-experimental and/or nonexperimental data (both of which lack such clarity). Yet in the author’s review of 64 recent meta-analytic articles, 11 were found to improperly make such aggregations. Why? The author shows how the guidance provided by the leading proponents of meta-analysis either blurs the distinction or is misleading.
Get full access to this article
View all access options for this article.
References
1.
Baltes, B. B., T. E. Briggs, J. W. Huff, J. A. Wright, and G. A. Neuman. 1999. Flexible and compressed workweek schedules: A meta-analysis. Journal of Applied Psychology84 (4): 496-513.
2.
Berk, R. A., and D. A. Freedman. 2002. Statistical assumptions as empirical commitments. In Law, punishment, and social control: Essays in honor of Sheldon Messinger (2d ed.), edited by T. Blomberg and S. Cohen, 1-24. Hawthorne, NY: Aldine de Gruyter.
3.
Boruch, R.1997. Randomized experiments for planning and evaluation. Thousand Oaks, CA: Sage.
4.
Cameron, J., K. M. Banko, and W. D. Pierce. 1994. Pervasive negative effects of rewards on intrinsic motivation: The myth continues. Behavior Analyst24 (1): 1-44.
5.
Campbell, D. T., and J. C. Stanley. 1966. Experimental and quasi-experimental designs for research. Chicago: Rand McNally.
6.
Chalmers, T. C., H. Smith, B. Blackburn, B. Silverman, B. Schroeder, D. Reitman, and A. Ambroz. 1981. A method for assessing the quality of a randomized control trial. Controlled Clinical Trials2:31-49.
7.
Clark, M., and A. D. Oxman. (Eds.). 2001. Cochrane Reviewers’ Handbook 4.1.1 [Updated October 2001]. In: The Cochrane Library, Issue 4, 2001.
8.
Cook, T. D., and D. T. Campbell. 1979. Quasi-experimentation: Design and analysis issues for field settings. Boston: Houghton Mifflin.
9.
Cooper, H., and L. V. Hedges. 1994. Potentials and limitations of research synthesis. In The handbook of research synthesis, edited by H. Cooper and L. V. Hedges, 521-529. New York: Russell Sage Foundation.
10.
General Accounting Office. 1991. Designing evaluations. Program Evaluation and Methodology Division, May, United States General Accounting Office, PEMD-10.1.4.
11.
Glass, G. V.1976. Primary, secondary, and meta-analysis. Educational Researcher5:3-8.
12.
Glass, G. V.1978. In defense of generalization. The Behavioral and Brain Sciences3:394-395.
13.
Glass, G. V., B. McGaw, and M. L. Smith. 1981. Meta-analysis in social research. Beverly Hills, CA: Sage.
14.
Hedges, L. V., and I. Olkin. 1985. Statistical methods for meta-analysis. Orlando, FL: Academic Press.
15.
Hunter, J. E., and F. L. Schmidt. 1990. Methods of meta-analysis: Correcting error and bias in research findings. Newbury Park, CA: Sage.
16.
Hunter, J. E., F. L. Schmidt, and G. B. Jackson. 1982. Meta-analysis: Cumulating research findings across studies. Beverly Hills, CA: Sage.
17.
Jadad, A. R., R. A. Moore, D. Carroll, C. Jenkinson, D. J. M. Reynolds, D. J. Gavaghan, and H. J. McQuay. 1996. Assessing the quality of randomized clinical trials: Is blinding necessary?Controlled Clinical Trials17:1-12.
18.
Jenkins, G.D., Jr., A. Mitra, N. Gupta, and J. D. Shaw. 1998. Are financial incentives related to performance? A meta-analytic review of empirical research. Journal of Applied Psychology83 (5): 777-787.
19.
Lee, D. M. 1969. Does Color Enhance Short Term Memory? Ph.D. diss., University of California, Los Angeles.
20.
Lepper, M., J. Henderlong, and I. Gingras. 1999. Understanding the effects of extrinsic rewards on intrinsic motivation—uses and abuses of meta-analysis: Comment on Deci, Koestner, and Ryan (1999). Psychological Bulletin125 (6): 669-676.
21.
Matt, G. E., and T. D. Cook. 1994. Threats to the validity of research synthesis. In The handbook of research synthesis, edited by H. Cooper and L. V. Hedges, 503-520. New York: Russell Sage Foundation.
22.
Orr, L. L.1999. Social experiments: Evaluating public programs with experimental methods. Thousand Oaks, CA: Sage.
23.
Ostroff, C., and D. A. Harrison. 1999. Meta-analysis, level of analysis, and best estimates of population correlations: Cautions for interpreting meta-analytic results in organizational behavior. Journal of Applied Psychology84 (2): 260-270.
24.
Porter, A. L., T. Connolly, R. G. Heikes, and C. Y. Park. 1981. Misleading indicators: The limitations of multiple linear regression in formulation of policy recommendations. Policy Sciences13:397-418.
25.
Robertson, P., and S. J. Seneviratne. 1996. Outcomes of planned organizational change in the public sector: A meta-analytic comparison to the private sector. Public Administration Review55 (6): 547-558.
26.
Rosenthal, R.1984. Meta-analytic procedures for social research. Beverly Hills, CA: Sage.
27.
Schmidt, F. L. & Hunter, J. E.1984. A within setting test of the situational specificity hypothesis in personnel selection. Personnel Psychology37: 317-326.
28.
Wortman, P. M.1994. Judging research quality. In The handbook of research synthesis, edited by H. Cooper and L. V. Hedges, 97-109. New York: Russell Sage Foundation.
29.
Wortman, P. M., and W. H. Yeaton. 1983. Synthesis of results in controlled trials of coronary artery bypass graft surgery. Evaluation Studies Review Annual8:536-551.
