Meta-analytic strategies are rapidly becoming the method of choice for researchers engaged in the review on topics spanning the scope of psychology. Although the assertion has been made that meta-analysis is an empirically reliable method of research synthesis, this claim has not been validated. To validate this claim, meta-analyses from three different topic areas, in each of which 3 or 4 reviewers carried out an independent meta-analysis is presented. Intraclass reliability coefficients and agreement rates were calculated and are used to support the assertion that meta-analysis is a reliable method for integration of research where the literature to be reviewed can be quantified and is amenable to meta-analytic techniques and analysis.
Get full access to this article
View all access options for this article.
References
1.
American Psychological Association. (1994) Publication manual of the American Psychological Association. (4th ed.) Washington, DC: American Psychological Association.
2.
BakanD. (1966) The test of significance in psychological research. Psychological Bulletin, 66, 1–29.
3.
BastaracheC. (1996) The Wisconsin Card Sorting Test: An indicator of schizophrenia?Unpublished Honors thesis, York Univer., Toronto, Canada.
4.
BemD. J. (1995) Writing a review article for Psychological Bulletin. Psychological Bulletin, 118, 172–177.
5.
CicchettiD. V.SparrowS. S. (1981) Developing criteria for establishing the interrater reliability of specific items in a given inventory. American Journal of Mental Deficiency, 86, 127–137.
6.
CohenJ. (1988) Statistical power analysis for the behavioral sciences. New York: Academic Press.
7.
CohenJ. (1994) The earth is round (p<.05). American Psychologist, 49, 997–1003.
8.
CooperH.HedgesL. V. (1994) The handbook of research synthesis. New York: The Russell Sage Found.
9.
CooperH. M. (1989) Integrating research: A guide for literature reviews. Newbury Park, CA: Sage.
10.
Del GiudiceT. (1996) A meta-analytic study: Selective attention and neuropsychological assessment in schizophrenia. Unpublished Honors thesis, York Univer., Toronto, Canada.
11.
EysenckH. J. (1995) Meta analysis squared—does it make sense?American Psychologist, 50, 110.
12.
FleissJ. L. (1981) Statistical methods for rates and proportions. (2nd ed.) New York: Wiley.
13.
GlassG. V. (1976) Primary, secondary, and meta-analysis. Educational Researcher, 5, 3–8.
14.
GlassG. V.McGawB.SmithM. L. (1981) Meta-analysis in social research. Beverly Hills, CA: Sage.
15.
GrimL. G.YarnoldP. R. (1995) Reading and understanding multivariate statistics. Washington, DC: American Psychological Association.
16.
HartleyK. A. (1996) Meta-analysis of Wisconsin Card Sorting Test results among schizophrenics. Unpublished Honors thesis, York Univer., Toronto, Canada.
17.
HeatonR. K.CheluneG. J.TalleyJ.KayG. G.CurtissG. (1993) Wisconsin Card Sorting Test manual–revised and expanded. Odessa, FL: Psychological Assessment Resources.
18.
HedgesL. V.OlkinI. (1985) Statistical methods of meta-analysis. New York: Academic Press.
19.
HeinrichsR. W.ZakzanisK. K. (in press) Neurocognitive deficit in schizophrenia: A quantitative review. Neuropsychology.
20.
HeronJ. (1996) The problem with primary research: A meta-analytic study of VBR and schizophrenia. Unpublished Honors thesis, York Univer., Toronto, Canada.
21.
HunterJ. E.SchmidtF. L. (1990) Methods of meta-analysis: Correcting error and bias in research findings. Newbury, CA: Sage.
22.
KnightG. P.FabesR. A.HigginsD. A. (1996) Concerns about drawing causal inferences from meta-analyses: An example in the study of gender differences in aggression. Psychological Bulletin, 119, 410–421.
23.
LicciardoneL. (1996) Lateral ventricle enlargement in schizophrenia: An updated meta-analysis of lateral ventricle: Brain ratio. Unpublished Honors thesis, York Univer., Toronto, Canada.
24.
LightR. J.PillemerD. B. (1984) Summing up: The science of reviewing research. Cambridge, MA: Harvard Univer. Press.
25.
LipseyM. W.WilsonD. B. (1993) The efficacy of psychological, educational, and behavioral treatment. American Psychologist, 12, 1181–1209.
26.
MeehlP. E. (1990) Why summaries of research on psychological theories are often uninterpretable. Psychological Reports, 66, 195–244.
27.
MisirM. (1996) A meta-analytic review of VBR in schizophrenia. Unpublished Honors thesis, York Univer., Toronto, Canada.
28.
MorphetK. A. (1996) Wisconsin Card Sorting Test performance in schizophrenia: A meta-analytic review. Unpublished Honors thesis, York Univer., Toronto, Canada.
29.
OrwinR. G. (1994) Evaluating coding decisions. In CooperH.HedgesL. V. (Eds.), The handbook of research synthesis. New York: Russel Sage Foundation.
30.
OrwinR. G.CordrayD. S. (1985) Effects of deficient reporting on meta-analysis: A conceptual framework and reanalysis. Psychological Bulletin, 97, 134–147.
31.
RosenthalR. (1991) Meta-analytic procedures for social research. Newbury Park, CA: Sage.
32.
RosenthalR. (1993) Cumulating evidence. In KerenG.LewisC. (Eds.), A handbook for data analysis in the behavioral sciences: Methodological issues. Hillsdale, NJ: Erlbaum.
SchmidtF. L. (1992) What do data really mean? Research findings, meta-analysis, and cumulative knowledge in psychology. American Psychologist, 47, 1173–1181.
35.
SchmidtF. L. (1996) Statistical significance testing and cumulative knowledge in psychology: Implications for training of researchers. Psychological Methods, 1, 115–129.
36.
SmithM. L.GlassG. V. (1977) “Meta-analysis of psychotherapy outcome studies.”American Psychologist, 32, 752–760.
37.
SohnD. (1997) Questions for meta-analysis. Psychological Reports, 81, 3–15.
38.
StesselL. (1996) Dichotic listening in schizophrenics: A meta-analysis. Unpublished Honors thesis, York Univer., Toronto, Canada.
39.
StockW. A. (1994) Systematic coding for research synthesis. In CooperH.HedgesL. V. (Eds.), The handbook of research synthesis. New York: Russel Sage Found.
40.
TaveggiaT. C. (1974) Resolving research controversy through empirical cumulation: Toward reliable sociological knowledge. Sociological Methods and Research, 2, 395–407.
41.
WolfF. M. (1986) Meta-analysis: Quantitative methods for research synthesis. Newbury Park, CA: Sage.
