Researchers are becoming interested in combining meta-analytic techniques and structural equation modeling to test theoretical models from a pool of studies. Most existing procedures are based on the assumption that all correlation matrices are homogeneous. Few studies have addressed what the next step should be when studies being analyzed are heterogeneous and the search for moderator variables for homogeneous subgroup analysis fails. Cluster analysis is proposed and evaluated in this article as an exploratory tool to classify studies into relatively homogeneous groups. Simulation studies indicate that using Euclidean distance on raw correlation coefficients or U-transformed scores with the complete linkage or Ward’s minimum-variance methods will provide satisfactory results.
Aldenderfer, M. S., & Blashfield, R. K. (1984). Cluster analysis. Beverly Hills, CA: Sage.
2.
Bacon, D. R. (2001). An evaluation of cluster analytic approaches to initial model specification. Structural Equation Modeling, 8, 397-429.
3.
Becker, B. J. (1992). Using results from replicated studies to estimate linear models. Journal of Educational Statistics, 17, 341-362.
4.
Becker, B. J. (1995). Corrections to “Using results from replicated studies to estimate linear models.”Journal of Educational Statistics, 20, 100-102.
5.
Becker, B. J. (2000). Multivariate meta-analysis. In H. E. A. Tinsley & S. D. Brown (Eds.) Handbook of applied multivariate statistics and mathematical modeling (pp.499-525). San Diego, CA: Academic Press.
6.
Becker, B. J. (2001). Examining theoretical models through research synthesis: The benefits of model-driven meta-analysis. Evaluation and the Health Professions, 24, 190-217.
7.
Becker, B. J., & Schram, C. M. (1994). Examining explanatory models through research synthesis. In H. Cooper & L. V. Hedges (Eds.), The handbook of research synthesis (pp. 357-381). New York: Russell Sage.
8.
Blashfield, R. K., & Aldenderfer, M. S. (1989). The methods and problems of cluster analysis. In J. R. Nesselroade & R. B. Cattell (Eds.), Handbook of multivariate experimental psychology (pp. 447-473). New York: Plenum.
9.
Brown, S. P., & Peterson, R. A. (1993). Antecedents and consequences of salesperson job satisfaction: meta-analysis and assessment of causal effects. Journal of Marketing Research, 30, 63-77.
10.
Brown, S. P., & Stayman, D. M. (1992). Antecedents and consequences of attitude toward the ad: A meta-analysis. Journal of Consumer Research, 19, 34-51.
11.
Cheung, M. W. L., & Chan, W. (2005). Meta-analytic structural equation modeling: A two-stage approach. Psychological Methods, 10, 40-64.
12.
Cheung, S. F. (2000). Examining solutions to two practical issues in meta-analysis: Dependent correlations and missing data in correlation matrices (Doctoral dissertation, Chinese University of Hong Kong, 2000). Dissertation Abstracts International, 61(8-B), 4469.
13.
Cohen, J. (1960). A coefficient of agreement for normal scales. Educational and Psychological Measurement, 20, 37-46.
14.
Colquitt, J. A., LePine, J. A., & Noe, R. A. (2000). Toward an integrative theory of training motivation: A meta-analytic path analysis of 20 years of research. Journal of Applied Psychology, 85, 678-707.
15.
Cortina, J. M. (2003). Apples and oranges (and pears, oh my!): The search for moderators in meta-analysis. Organizational Research Methods, 6, 415-439.
16.
Digman, J. M. (1997). Higher-order factors of the Big Five. Journal of Personality and Social Psychology, 73, 1246-1256.
17.
Dumenci, L., & Windle, M. (2001). Cluster analysis as a method of recovering types of intraindividual growth trajectories: A Monte Carlo study. Multivariate Behavioral Research, 36, 501-522.
18.
Engels, E. A., Schmid, C. H., Terrin, N., Olkin, I., & Lau, S. (2000). Heterogeneity and statistical significance in meta-analysis: An empirical study of 125 meta-analyses. Statistics in Medicine, 19, 1707-1728.
19.
Everitt, B. S., Landau, S., & Leese, M. (2001). Cluster analysis (4th ed.). London: Oxford University Press.
20.
Field, A. P. (2001). Meta-analysis of correlation coefficients: A Monte Carlo comparison of fixed- and random-effects methods. Psychological Methods, 6, 161-180.
21.
Glass, G. V (1976). Primary, secondary, and meta-analysis of research. Educational Researcher, 5, 3-8.
22.
Glasziou, P. P., & Sanders, S. L. (2002). Investigating causes of heterogeneity in systematic reviews. Statistics in Medicine, 21, 1503-1511.
23.
Gordon, A. D. (1998).Cluster validation. In C. Hayashi, N. Ohsumi, K. Yajima, Y. Tanaka, H.-H. Bock, & Y. Baba (Eds.), Data science: Classification and related methods (pp. 22-39). New York: Springer-Verlag.
24.
Gordon, A. D. (1999). Classification (2nd ed.). London: Chapman & Hall.
25.
Hafdahl, A. R. (2002). Multivariate meta-analysis for exploratory factor analytic research (Doctoral dissertation, University of North Carolina at Chapel Hill, 2002). Dissertation Abstracts International, 62(8-B), 3843.
26.
Hedges, L. V., & Olkin, I. (1983). Clustering estimates of effect magnitude from independent studies. Psychological Bulletin, 93(3), 563-573.
27.
Hedges, L. V., & Olkin, I. (1985). Statistical methods for meta-analysis. Orlando, FL: Academic Press.
28.
Hedges, L. V., & Vevea, J. L. (1998). Fixed- and random-effects models in meta-analysis. Psychological Methods, 3, 486-504.
29.
Hubert, L., & Arabie, P. (1985). Comparing partitions. Journal of Classification, 2, 193-218.
30.
Hunter, J. E., & Schmidt, F. L. (1990). Methods of meta-analysis: Correcting error and bias in research findings. Newbury Park, CA: Sage.
31.
Jardine, N., & Sibson, R. (1971). Mathematical taxonomy. New York: John Wiley.
32.
Kalaian, H. A., & Raudenbush, S. W. (1996). A multivariate mixed linear model for meta-analysis. Psychological Methods, 1, 227-235.
33.
Lance, G. N., & Williams, W. T. (1967). A general theory of classificatory sorting strategies. Computer Journal, 9, 373-380.
34.
Lijmer, J. G., Bossuyt, P. M. M., & Heisterkamp, S. H. (2002). Exploring sources of heterogeneity in systematic reviews of diagnostic tests. Statistics in Medicine, 21, 1525-1537.
35.
Mardia, K. V., Kent, J. T., & Bibby, J. M. (1979). Multivariate analysis. London: Academic Press.
36.
Marín-Martínez, F., & Sánchez-Meca, J. (1998). Testing for dichotomous moderators in meta-analysis. Journal of Experimental Education, 67, 69-81.
37.
Miller, N., & Pollock, V. E. (1994). Meta-analytic synthesis for theory development. In H. Cooper & L. V. Hedges (Eds.), The handbook of research synthesis (pp. 457-483). New York: Russell Sage.
38.
Milligan, G. W., & Cooper, M. C. (1985). An examination of procedures for determining the number of cluster in a data set. Psychometrika, 50, 159-179.
39.
Milligan, G. W., & Cooper, M. C. (1986). A study of the comparability of external criteria for hierarchical cluster analysis. Multivariate Behavioral Research, 21, 41-58.
40.
Milligan, G. W., & Cooper, M. C. (1987). Methodology review: Clustering methods. Applied Psychological Measurement, 11, 329-354.
41.
Milligan, G. W., & Cooper, M. C. (1988). A study of standardization of variables in cluster analysis. Journal of Classification, 5, 181-204.
42.
Milligan, G. W., & Isaac, P. (1980). The validation of four ultrametric clustering algorithms. Pattern Recognition, 12, 41-50.
43.
Muthén, B. (2004). Latent variable analysis: Growth mixture modeling and related techniques for longitudinal data. In D. Kaplan (Ed.), Handbook of quantitative methodology for the social sciences (pp. 345-368). Newbury Park, CA: Sage.
44.
Muthén, B., Brown, C. H., Masyn, K., Jo, B., Khoo, S. T., Yang, C. C., et al. (2002). General growth mixture modeling for randomized preventive interventions. Biostatistics, 3, 459-475.
45.
National Research Council. (1992). Combining information: Statistical issues and opportunities for research. Washington, DC: National Academy Press.
46.
Overall, J. E., & Magee, K. N. (1992). Replication as a rule for determining the number of clusters in hierarchical cluster analysis. Applied Psychological Measurement, 16, 119-128.
47.
Pigott, T. D. (2001). Missing predictors in models of effect size. Evaluation and the Health Professionals, 24, 277-307.
48.
Poon, W. Y., Chan, W., Lee, S. Y., & Leung, K. (1993). Preliminary analysis of multiple group structural equation modeling via cluster analysis. In American Statistical Association (Ed.), Proceedings of the Social Statistics Section (pp. 368-373). Washington, DC: American Statistical Association.
49.
Raudenbush, S. W., Becker, B. J., & Kalaian, H. (1988). Modeling multivariate effect sizes. Psychological Bulletin, 103, 111-120.
50.
Roussos, L. A., Stout, W. F., & Marden, J. I. (1998). Using new proximity measures with hierarchical cluster analysis to detect multidimensionality. Journal of Educational Measurement, 35, 1-30.
51.
SAS Institute. (1995). SAS/IML software: Changes and enhancements, through Release 6.11. Cary, NC: Author.
52.
Shadish, W. R. (1996). Meta-analysis and the exploration of causal mediating processes: A primer of examples, methods, and issues. Psychological Methods, 1, 47-65.
53.
Sokal, R. R., & Michener, C. D. (1958). A statistical method for evaluating systematic relationships. University of Kansas Science Bulletin, 38, 1409-1438.
54.
Sokal, R. R., & Sneath, P. H. (1963). Principles of numerical taxonomy. London: Freeman.
55.
Song, F., Sheldon, T. A., Sutton, A. J., & Jones, D. R. (2001). Methods for exploring heterogeneity in meta-analysis. Evaluation and the Health Professions, 24, 126-151.
56.
Verhaeghen, P., & Salthouse, T. A. (1997). Meta-analyses of age-cognition relations in adulthood: Estimates of linear and nonlinear age effects and structural models. Psychological Bulletin, 122, 231-249.
57.
Viswesvaran, C., & Ones, D. S. (1995). Theory testing: Combining psychometric meta-analysis and structural equations modeling. Personnel Psychology, 48, 865-885.
58.
Viswesvaran, C., & Sanchez, J. I. (1998). Moderator search in meta-analysis: A review and cautionary note on existing approaches. Educational and Psychological Measurement, 58, 77-87.
