In its short history meta-analysis has been the object of both high praise and pointed criticism. This paper explores the conceptual bases of the controversy. Meta-analytic and traditional research reviews are compared on 4 dimensions: scientific rigor, detecting small effects, problems of oversimplification, and potential policy impact. It is argued that choice of procedures should be determined by the specific purpose of a review rather than by canon. The author cites the benefits of viewing meta-analysis as a flexible arsenal of quantitative tools rather than a unitary method. Several future goals for the developing science of research synthesis are discussed.
Get full access to this article
View all access options for this article.
References
1.
Campbell, D.T.Degrees of freedom and the case study. Comparative Political Studies, 1975, 8, 178-193.
2.
Gohen, J.Statistical power analysis for the behavioral sciences. New York: Academic Press, 1969.
3.
Cook, T.D., & Leviton, L.C.Reviewing the literature: A comparison of traditional methods with meta-analysis . Journal of Personality, 1980 , 48, 449-472.
Cooper, H.M., & Rosenthal, R.Statistical versus traditional procedures for summarizing research findings . Psychological. Bulletin, 1980, 87, 442-449.
6.
Cotton, J.L., & Cook, M.S.Meta-analysis and the effects of various reward systems: Some different conclusions from Johnson et alPsychological Bulletin, 1982, 92, 176-183.
7.
Devine, E.C., & Cook, T.D.Effects of psycho-educational interventions on length of hospital stay: A meta-analytic review of 34 studies. In R. J. Light (Ed.), Evaluation studies review annual, (Vol. 8). Beverly Hills, Calif.: Sage, 1983.
8.
Eagly, A.H., & Carli, L.L.Sex of researchers and sex-typed communications as determinants of sex differences in influenceability : A meta-analysis of social influence studies. Psychological Bulletin, 1981, 90, 1-20.
9.
Educational Research Service.Class size research : A critique of recent meta-analyses. Phi Delta Kappan, 1980 (December), 239-241.
10.
Eysenck, H.J.An exercise in mega-silliness. American Psychologist , 1978, 33, 517.
Gilbert, J.P., Light, R.J., & Mosteller, F.Assessing social innovations: An empirical base for policy. In C. A. Bennett & A. A. Lumsdaine (Eds.), Evaluation and experiment. New York: Academic Press, 1975.
13.
Glass, G.V.Primary, secondary and meta-analysis of research. Educational Researcher, 1976, 5, 3-8.
14.
Glass, G.V.On criticism of our class size/ student achievement research: No points conceded . Phi Delta Kappan, 1980 (December), 242-244.
15.
Glass, G.V., McGaw, B., & Smith, M.L.Meta-analysis of social research. Beverly Hills, Calif. : Sage, 1981.
16.
Glass, G.V., & Smith, M.L.Reply to Eysenck. American Psychologist, 1978 , 33, 517.
17.
Glass, G.V. , & Smith, M.L.Meta-analysis of research on class size and achievement. Educational Evaluation and Policy Analysis, 1979, 1, 2-16.
18.
Gottman, J.J., & Glass, G.V.Analysis of interrupted time-series experiments. In T. R. Kratochwill (Ed.), Single subject research . New York: Academic Press, 1978.
19.
Greenwald, A.G.Consequences of prejudice against the null hypothesis. Psychological Bulletin, 1975, 82, 1-20.
Hedges, L.V.Estimation of effect size from a series of independent experiments . Psychological Bulletin, 1982, 92, 490-499.
22.
Hedges, L.V., & Olkin, I.Vote-counting methods in research synthesis. Psychological Bulletin, 1980, 88, 359-369.
23.
Jackson, G.B.Methods for integrative reviews. Review of Educational Research , 1980, 50, 438-460.
24.
Johnson, D.W., Maruyama, G., & Johnson, R.T.Separating ideology from currently available data: A reply to Cotton & Cook & McGlynn. Psychological Bulletin, 1982, 92, 186-192.
25.
Johnson, D.W., Maruyama, G., Johnson, R., Nelson, D., & Skon, L.Effects of cooperative, competitive, and individualistic goal structures on achievement: A meta-analysis. Psychological Bulletin, 1981, 89, 47-62.
26.
Jones, R.R.Weinrott, M.R., & Vaught, R.S.Effects of serial dependency on the agreement between visual and statistical inference. Journal of Applied Behavior Analysis, 1978, 11, 277-283.
27.
Kratochwill, T.R.N = 1: An alternative research strategy for school psychologists. Journal of School Psychology, 1977, 15, 239-249.
28.
Kulik, J.A., Kulik, C.C., & C ohen, P.A.A meta-analysis of outcome studies of Keller's personalized system of instruction . American Psychologist, 1979 , 34, 307-318.
29.
Kulik, J.A., Kulik, C.C., & Cohen, P.A.Effectiveness of computer-based college teaching: A meta-analysis of findings . Review of Educational Research, 1980, 50, 525-544.
30.
Leviton, L.C., & Cook, T.D.What differentiates meta-analysis from other forms of review. Journal of Personality, 1981, 49, 231-236.
31.
Light, R.J.Synthesis methods: Some judgment calls that must be made. Evaluation in Education, 1980, 4, 18-21.
32.
Light, R.J., & Pillemer, D.B.Numbers and narrative: Combining their strengths in research reviews. Harvard Educational Review, 1982, 52, 1-26.
33.
Light, R.J., & Smith, P.V.Accumulating evidence: Procedures for resolving contradictions among different research studies. Harvard Educational Review, 1971, 41, 429-471.
34.
Madey, D.L.Some benefits of integrating qualitative and quantitative methods in program evaluation, with illustrations. Educational Evaluation and Policy Analysis, 1982, 4, 223-236.
35.
McGlynn, R.P.A comment on the meta-analysis of goal structures. Psychological Bulletin, 1982, 92, 184-185.
36.
Meehl, P.E.Seer over sign: The first good example. Journal of Experimental Research in Personality, 1965, 1, 27-32.
37.
Pillemer, D.B., & Light, R.J.Benefiting from variation in study outcomes. New Directions for Methodology of Social and Behavioral Science, 1980, 5, 1-12. (a)
38.
Pillemer, D.B., & Light, R.J.Synthesizing outcomes: How to use research evidence from many studies . Harvard Educational Review, 1980, 50, 176-195. (b)
39.
Presby, S.Overly broad categories obscure important differences between therapies . American Psychologist, 1978, 33, 514-515.
40.
Reichardt, C. S., & Cook, T. D.Beyond qualitative versus quantitative methods. In T. D. Cook & C. S. Reichardt (Eds.), Qualitative and quantitative methods in evaluation research. Beverly Hills, Calif.: Sage, 1979.
41.
Rein, M., & White, S.H.Can policy research help policy?The Public Interest , 1977, No. 49, 119-136.
Rosenthal, R.Assessing the statistical and social importance of the effects of psychotherapy . Journal of Consulting and Clinical Psychology , 1983, 51, 4-13.
44.
Rosenthal, R., & Rubin, D.B.Interpersonal expectancy effects: The first 345 studies. Behavioral and Brain Sciences, 1978, 3, 377-415.
45.
Rosenthal, R., & Rubin, D.B.A simple, general purpose display of magnitude of experimental effect . Journal of Educational Psychology, 1982 , 74, 166-169. (a)
46.
Rosenthal, R., & Rubin, D.B.Comparing effect sizes of independent studies. Psychological Bulletin, 1982, 92, 500-504. (b)
47.
Rosenthal, R., & Rubin, D.B.Further meta-analytic procedures for assessing cognitive gender differences . Journal of Educational Psychology, 1982 , 74, 708-712. (c)
48.
Smith, M.L.Publication bias and meta-analysis. Evaluation in Education , 1980, 4, 22-24.
49.
Smith, M.L., & Glass, G.V.Meta-analysis of psychotherapy outcome studies. American Psychologist , 1977, 32, 752-760.
50.
Smith, M.S., & Bissell, J.The impact of Head Start. Harvard Educational Review , 1970, 40, 51-104.
51.
Tukey, J.W.The future of data analysis. The Annals of Mathematical Statistics , 1962, 33, 1-67.
52.
Walberg, H.J.Synthesis of research on teaching. In M. C. Wittrock (Ed.), The third handbook of research on teaching . Washington, D.C.: American Educational Research Association, 1983.
53.
Wortman, P.M.Meta-analysis: A validity perspective. Paper presented at the Evaluation Research Society, Baltimore, October 1982.
