This article conceptually describes the steps in conducting quantitative meta-analyses of counseling psychology research with minimal reliance on statistical formulas. The authors identify sources that describe necessary statistical formula for various meta-analytic calculations and describe recent developments in meta-analytic techniques. The authors illustrate meta-analytic procedures with recently published meta-analyses in counseling psychology, and last, they provide a guide for writing up meta-analytic studies for publication.
Get full access to this article
View all access options for this article.
References
1.
Ahn, H., & Wampold, B. E. (2001). Where or where are the specific ingredients? A meta-analysis of component studies in counseling and psychotherapy. Journal of Counseling Psychology, 48, 251-257.
2.
Allumbaugh, D. L., & Hoyt, W. T. (1999). Effectiveness of grief therapy: A meta-analysis. Journal of Counseling Psychology, 46, 370-380.
3.
Barber, J. P., & Milrod, B. (2004). Pitfalls of meta-analyses. American Journal of Psychiatry, 161, 1131-1131.
4.
Baskin, T. W., Tierney, S. C., Minami, T., & Wampold, B. E. (2003). Establishing specificity in psychotherapy: A meta-analysis of structural equivalence of placebo controls. Journal of Consulting and Clinical Psychology, 71, 973-979.
5.
Behrens, J. T. (1997). Does the White Racial Identity Attitude Scale measure racial identity?Journal of Counseling Psychology, 44, 3-12.
6.
Borenstein, M. (2001). Comprehensive meta-analysis [Computer software]. Englewood, NJ: Biostat.
7.
Bowman, D., Scogin, F., Floyd, M., & Mckendree-Smith, N. (2001). Psychotherapy length of stay and outcome: A meta-analysis of the effect of therapist sex. Psychotherapy: Theory, Research, Practice, Training, 38, 142-148.
8.
Cheung, S. F., & Chan, D. K. S. (2004). Dependent effect sizes in meta-analysis: Incorporating the degree of interdependence. Journal of Applied Psychology, 89, 780-791.
9.
Chinman, M. J., Rosenheck, R. A., & Lam, J. A. (2000). Client-case manager racial matching in a program for homeless persons with serious mental illness. Psychiatric Services, 51, 1265-1272.
10.
Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum.
11.
Cohen, J. (1994). The earth is round (p < .05). American Psychologist, 12, 997-1003.
12.
Cohn, L. D., & Becker, B. J. (2003). How meta-analysis increases statistical power. Psychological Methods, 8, 243-253.
13.
Coleman, H. L. K., Wampold, B. E., & Casali, S. L. (1995). Ethnic minorities’ ratings of ethnically similar and European American counselors: A meta-analysis. Journal of Counseling Psychology, 42, 55-64.
14.
Conn, V. S., Valentine, J. C., Cooper, H. M., & Rantz, M. J. (2003). Grey literature in meta-analyses. Nursing Research, 52, 256-261.
15.
Donner, A., Piaggio, G., & Villar, J. (2003). Meta-analyses of cluster randomization trials: Power considerations. Evaluation and the Health Professions, 26, 340-351.
16.
Eagly, A. H., & Carli, L. L. (1981). Sex of researchers and sex-typed communications as determinants of sex differences in influenceability: A meta-analysis of social influence studies. Psychological Bulletin, 90, 1-20.
17.
Eysenck, H. J. (1952). The effects of psychotherapy: An evaluation. Journal of Consulting Psychology, 16, 319-324.
18.
Eysenck, H. J. (1978). An exercise in mega-silliness. American Psychologist, 33, 517-517.
19.
Fagley, N. S. (1985). Applied statistical power analysis and the interpretation of nonsignifi-cant results by research consumers. Journal of Counseling Psychology, 22, 391-398.
20.
Fleiss, J. L. (1994). Measures of effect size for categorical data. In H. Cooper & L. V. Hedges (Eds.), The handbook of research synthesis (pp. 245-260). New York: Russell Sage.
21.
Friedman, H. (1968). Magnitude of experimental effect and a table for its rapid estimation. Psychological Bulletin, 70, 245-251.
22.
Gillett, R. (2003). The metric comparability of meta-analytic effect-size estimators from factorial design. Psychological Methods, 8, 419-433.
23.
Gleser, L. J., & Olkin, I. (1994). Stochastically dependent effect sizes. In H. Cooper & L. V. Hedges (Eds.), The handbook of research synthesis (pp. 339-355). New York: Russell Sage.
24.
Haase, R. F., Ellis, M. V., & Ladany, N. (1989). Multiple criteria for evaluating the magnitude of experimental effects. Journal of Counseling Psychology, 36, 511-516.
25.
Haase, R. F., Waechter, D. M., & Solomon, G. S. (1982). How significant is a significant difference? Average effect size of research in counseling psychology. Journal of Counseling Psychology, 29, 58-65.
26.
Hall, J. A., Rosenthal, R., Tickle-Degnen, L., & Mosteller, F. (1994). Hypotheses and problems in research synthesis. In H. Cooper & L. V. Hedges (Eds.), The handbook of research synthesis (pp. 17-28). New York: Russell Sage.
27.
Hedges, L. V. (1987). How hard is hard science, how soft is soft science? The empirical cumulativeness of research. American Psychologist, 42, 443-455.
28.
Hedges, L. V. (1994). Fixed effects models. In H. Cooper & L. V. Hedges (Eds.), The handbook of research synthesis (pp. 285-299). New York: Russell Sage.
29.
Hedges, L. V., & Olkin, I. (1980). Vote-counting methods in research synthesis. Psychological Bulletin, 88, 359-369.
30.
Hedges, L. V., & Olkin, I. (1985). Statistical methods for meta-analysis. San Diego, CA: Academic Press.
31.
Hedges, L. V., & Pigott, T. D. (2001). The power of statistical tests in meta-analysis. Psychological Methods, 6, 203-217.
32.
Hedges, L. V., & Pigott, T. D. (2004). The power of statistical tests for moderators in meta-analysis. Psychological Methods, 9, 426-445.
33.
Hedges, L. V., & Vevea, J. L. (1998). Fixed- and random-effects models in meta-analysis. Psychological Methods, 3, 486-504.
34.
Helms, J. E., & Carter, R. T. (1990). Development of the White Racial Identity Attitude Inventory. In J. E. Helms (Ed.), Black and White racial identity: Theory, research, and practice (pp. 67-80). Westport, CT: Greenwood.
35.
Helms, J. E., Henze, K. T., Sass, T. L., & Mifsud, V. A. (2006). Treating Cronbach’s alpha reliability coefficients as data in counseling research. The Counseling Psychologist, 34, 630-660.
36.
Henson, R. K. (2006). Effect-size measures and meta-analytic thinking in counseling psychology research. The Counseling Psychologist, 34, 601-629.
37.
Holland, J. L. (1985). Making of vocational choices: A theory of vocational personalities and work environments (2nd ed.). Englewood Cliffs, NJ: Prentice Hall.
38.
Holloway, E. L., & Wampold, B. E. (1986). Relation between conceptual level and counseling-related tasks: A meta-analysis. Journal of Counseling Psychology, 33, 310-319.
39.
Hoyt, W. T. (1996). Antecedents and effects of perceived therapist credibility: A meta-analysis. Journal of Counseling Psychology, 43, 430-447.
40.
Hoyt, W. T., Warbasse, R. E., & Chu, E. Y. (2006). Construct validation in counseling psychology research. The Counseling Psychologist, 34, 769-805.
41.
Hsu, L. M. (2002). Fail-Safe Ns for one versus two tailed tests lead to different conclusions about publication bias. Understanding Statistics, 1, 85-100.
42.
Hunter, J. E., & Schmidt, F. L. (1990). Methods of meta-analysis: Correcting error and bias in research findings. London: Sage.
43.
Hunter, J. E., & Schmidt, F. L. (1994). Correcting for sources of artificial variation across studies. In H. Cooper & L. V. Hedges (Eds.), The handbook of research synthesis (pp. 323-336). New York: Russell Sage.
44.
Iyengar, S., & Greenhouse, J. B. (1988). Selection models and the file drawer problem. Statistical Science, 3, 109-135.
45.
Johnson, B. T., & Eagly, A. H. (1989). Effects of involvement on persuasion: A meta-analysis. Psychological Bulletin, 106, 290-314.
46.
Kahn, J. H. (2006). Factor analysis in counseling psychology research, training, and practice: Principles, advances, and applications. The Counseling Psychologist, 34, 684-718.
47.
Keef, S. P., & Roberts, L. A. (2004). The meta-analysis of partial effect sizes. British Journal of Mathematical and Statistical Psychology, 57, 97-129.
48.
Kraemer, H. C., Gardner, C., Brooks, J. O., & Yesavage, J. A. (1998). Advantage of excluding underpowered studies in meta-analyses: Inclusionist versus exclusionist viewpoints. Psychological Methods, 3, 23-31.
49.
Landman, J. T., & Dawes, R. M. (1982). Psychotherapy outcome: Smith and Glass’ conclusions stand up under scrutiny. American Psychologist, 37, 504-516.
50.
Lipsey, M. W. (1994). Identifying potentially interesting variables and analysis opportunities. In H. Cooper & L. V. Hedges (Eds.), The handbook of research synthesis (pp. 111-123). New York: Russell Sage.
51.
Lipsey, M. W., & Wilson, D. B. (1993). The efficacy of psychological, educational, and behavioral treatment: Confirmation from meta-analysis. American Psychologist, 48, 1181-1209.
52.
Lipsey, M. W., & Wilson, D. B. (2001). Practical meta-analysis. Thousand Oaks, CA: Sage.
53.
Marascuilo, L. A., & Serlin, R. C. (1988). Statistical methods for the social and behavioral sciences. New York: Freeman.
54.
Martens, M. P., & Haase, R. F. (2006). Advanced applications of structural equation modeling in counseling psychology research. The Counseling Psychologist, 34, 878-911.
55.
McCullough, M. E. (1999). Research on religion-accommodative counseling: Review and meta-analysis. Journal of Counseling Psychology, 46, 92-98.
56.
Multon, K. D., Brown, S. D., & Lent, R. W. (1991). Relation of self-efficacy beliefs to academic outcomes: A meta-analytic investigation. Journal of Counseling Psychology, 38, 30-38.
57.
Muncer, S. J., Craigie, M., & Holmes, J. (2003). Meta-analysis and power: Some suggestions for the use of power in research synthesis. Understanding Statistics, 2, 1-12.
58.
Newell, M. L. (in press). African American students and aggressive behaviors: Situating intervention effectiveness within the context of discourse. School Psychology Quarterly.
59.
Olejnik, S., & Algina, J. (2003). Generalized eta and omega squared statistics: Measures of effect size for some common research designs. Psychological Methods, 8, 434-447.
60.
Orwin, R. G. (1994). Evaluating coding decisions. In H. Cooper & L. V. Hedges (Eds.), The handbook of research synthesis (pp. 301-321). New York: Russell Sage.
61.
Paquin, M. J. R. (1983). Beyond significant yet meaningless results in psychotherapy research. Psychotherapy: Theory, Research, and Practice, 20, 38-40.
62.
Peterson, R. A., & Brown, S. P. (2005). On the use of beta-coefficients in meta-analysis. Journal of Applied Psychology, 90, 175-181.
63.
Quintana, S. M., Troyano, N., & Taylor, G. (2001). Cultural validity and inherent challenges in quantitative methods for multicultural research. In J. G. Ponterotto, J. M. Casas, L. A. Suzuki, & C. M. Alexander (Eds.), Handbook of multicultural counseling (2nd ed., pp. 604-630). Thousand Oaks, CA: Sage.
64.
Raudenbush, S. W. (1994). Random effects models. In H. Cooper & L. V. Hedges (Eds.), The handbook of research synthesis (pp. 301-321). New York: Russell Sage.
65.
Rosenberg, M., Adams, D. C., & Gurevitch, J. (1999). MetaWin: Statistical software for meta-analysis with resampling tests (Version 2.0) [Computer software]. Sunderland, MA: Sinauer Associates.
66.
Rosenthal, R. (1979). The “file drawer problem” and tolerance for null results. Psychological Bulletin, 86, 638-641.
67.
Rosenthal, R. (1990). How are we doing in soft psychology?American Psychologist, 45, 775-776.
68.
Rosenthal, R. (1991). Meta-analystic procedures for social research. Newbury Park, CA: Sage.
69.
Rosenthal, R. (1994). Parametric measures of effect size. In H. Cooper & L. V. Hedges (Eds.), The handbook of research synthesis (pp. 231-244). New York: Russell Sage.
70.
Rosenthal, R. (1995). Writing meta-analytic reviews. Psychological Bulletin, 118, 183-192.
71.
Rosenthal, R., & Rubin, D. B. (1982). A simple, general purpose display of magnitude of experimental effect. Journal of Educational Psychology, 71, 166-169.
72.
Rosenthal, R., & Rubin, D. B. (2003). r equivalent: A simple effect size indicator. Psychological Methods, 8, 492-496.
73.
Rosnow, R. L., & Rosenthal, R. (2003). Effect sizes for experimenting psychologists. Canadian Journal of Experimental Psychology, 57, 221-237.
74.
Rosnow, R. L., Rosenthal, R., & Rubin, D. B. (2000). Contrasts and correlations in effect-size estimation. Psychological Science, 11, 446-453.
75.
Rounds, J., & Tracey, T. J. (1996). Cross-cultural structural equivalence of RIASEC models and measures. Journal of Counseling Psychology, 43, 310-329.
76.
Sánchez-Meca, J., Marín-Martínez, F., & Chacón-Moscoso, S. (2003). Effect-size indices for dichotomized outcomes in meta-analysis. Psychological Methods, 8, 448-467.
77.
Schmidt, F. L., & Hunter, J. E. (1996). Measurement error in psychological research: Lessons from 26 research scenarios. Psychological Methods, 1, 199-223.
78.
Serlin, R. C., Wampold, B. E., & Levin, J. R. (2003). Should providers of treatment be regarded as a random factor? If it ain’t broke, don’t “fix” it: A comment on Siemer and Joorman (2003). Psychological Methods, 8, 524-534.
79.
Shadish, W. R., Cook, T. D., & Campbell, D. T. (2002). Experimental and quasi-experimental designs for generalized causal inference. Boston: Houghton Mifflin.
80.
Shadish, W. R., & Haddock, C. K. (1994). Combining estimates of effect size. In H. Cooper & L. V. Hedges (Eds.), The handbook of research synthesis (pp. 261-281). New York: Russell Sage.
81.
Shadish, W. R., Jr., & Sweeney, R. B. (1991). Mediators and moderators in meta-analysis: There’s a reason we don’t let dodo birds tell us which psychotherapies should have prizes. Journal of Consulting and Clinical Psychology, 59, 883-893.
82.
Sharpe, D. (1997). Of apples and oranges, file drawers and garbage: Why validity issues in meta-analysis will not go away. Clinical Psychology Review, 17, 881-901.
83.
Sherry, A. (2006). Discriminant analysis in counseling psychology research. The Counseling Psychologist, 34, 661-683.
84.
Shin, S., Chow, C., Camacho-Gonsalves, T., Levy, R. J., Allen, I. E., & Leff, H. S. (2005). A meta-analytic review of racial-ethnic matching for African American and Caucasian American clients and clinicians. Journal of Counseling Psychology, 52, 45-56.
85.
Smith, M. L., & Glass, G. V. (1977). Meta-analysis of psychotherapy outcome studies. American Psychologist, 32, 752-760.
86.
Spengler, P., Strohmer, D., Dixon, D., & Shivy, V. (1995). A scientist-practitioner model of psychological assessment: Implications for training, practice and research. Counseling Psychologist, 23, 506-534
87.
Stock, W. A. (1994). Systematic coding for research synthesis. In H. Cooper & L. V. Hedges (Eds.), The handbook of research synthesis (pp. 125-138). New York: Russell Sage.
88.
Strube, M. J. (1988). Some comments on the use of magnitude-of-effect estimates. Journal of Counseling Psychology, 35, 342-345.
89.
Vacha-Haase, T., & Thompson, B. (2004). How to estimate and interpret various effect sizes. Journal of Counseling Psychology, 51, 473-481.
90.
Wampold, B. E. (2001). The great psychotherapy debate: Models, methods, and findings. Mahwah, NJ: Lawrence Erlbaum.
91.
Wampold, B. E., Minami, T., Baskin, T. W., & Tierney, S. C. (2002). A meta-(re)analysis of the effects of cognitive therapy versus ‘other therapies’ for depression. Journal of Affective Disorders, 68, 159-165.
92.
Wampold, B. E., Mondin, G. W., Moody, M., Stich, F., Benson, K., & Hyun-nie, A. (1997). A meta-analysis of outcome studies comparing bona fide psychotherapies: Empirically, “All must have prizes.”Psychological Bulletin, 122, 203-215.
93.
Wampold, B. E., & Serlin, R. C. (2000). The consequence of ignoring a nested factor on measures of effect size in analysis of variance. Psychological Methods, 5, 425-433.
94.
Wang, M. C., & Bushman, B. J. (1998). Using the normal quartile plot to explore meta-analytic data sets. Psychological Methods, 3, 46-54.
95.
Weston, R., & Gore, P. A., Jr. (2006). A brief guide to structural equation modeling. The Counseling Psychologist, 34, 719-751.
96.
Whiston, S. C., Sexton, T. L., & Lasoff, D. L. (1998). Career-intervention outcome: A replication and extension of Oliver and Spokane (1988). Journal of Counseling Psychology, 45, 150-165.
97.
White, H. D. (1994). Scientific communication and literature retrieval. In H. Cooper & L. V. Hedges (Eds.), The handbook of research synthesis (pp. 41-56). New York: Russell Sage.
98.
Worthington, R. L., & Whittaker, T. A. (2006). Scale development research: A content analysis and recommendations for best practices. The Counseling Psychologist, 34, 806-838.
99.
Wortman, P. M. (1994). Judging research quality. In H. Cooper & L. V. Hedges (Eds.), The handbook of research synthesis (pp. 97-109). New York: Russell Sage.
