Becker and Cote (1994) found that the correlated uniqueness model outperformed the confirmatory factor analysis and direct product models for multitrait-multimethod data. The present study analyzed20 multitrait-multirater performance appraisal matrices. The correlated uniqueness model was appropriate significantly more often than in Becker and Cote5 study and the other two models performed poorly. The proportions of trait and method variance in ratings were related to several rating system characteristics such as opportunity for raters to observe ratees and basing rating dimensions on a job analysis. Performance of all three models was better with larger proportions of trait variance and smaller proportions of method variance.
Get full access to this article
View all access options for this article.
References
1.
Albrecht, P.A., Glaser, E.M. & Marks, J. (1964). Validation of a multiple-assessment procedure for managerial personnel. Journal of Applied Psychology, 48: 351-360.
2.
Bagozzi, R.P. (1993). Assessing construct validity in personality research: Applications to measures of self-esteem. Journal of Research in Personality, 27: 49-87.
3.
Bagozzi, R.P. & Yi, Y. (1990). Assessing method variance in multitrait-multimethod matrices: The case of self-reported affect and perceptions of work. Journal of Applied Psychology, 75: 547-560.
4.
Bagozzi, R.P. & Yi, Y. (1991). Multitrait-multimethod matrices in consumer research. Journal of Consumer Research, 17: 426-439.
5.
Becker, T.E. & Cote, J.A. (1994). Additive and multiplicative method effects in applied psychological research: An empirical assessment of three models. Journal of Management, 20: 625-641.
6.
Becker, T.E. & Vance, R.J. (1993). Construct validity of three types of organizational citizenship behavior: An illustration of the direct product model with refinements. Journal of Management, 19: 663-682.
7.
Boomsma, A. (1985). Nonconvergence, improper solutions, and starting values in LISREL maximum likelihood estimation. Psychometrika, 50: 229-242.
8.
Barman, W.C. (1974). Rating of individuals in organizations: An alternate approach. Organizational Behavior and Human Performance, 12: 105-124.
9.
Borman, W.C. & Motowidlo, S.J. (1993). Expanding the criterion domain to include elements of contextual jerformance. Pp. 71-98 in N. Schmitt & W.C. Borman (Eds.), Personnel selection in organizations. San Francisco: Jossey-Bass.
10.
Boruch, R.F., Larkin, J.D., Wohns, L. & MacKinney, A.C. (1970). Alternative methods of analysis: Multitrait-multimethod data. Educational and Psychological Measurement, 30: 833-853.
11.
Brannick, M.T. & Spector, P.E. (1990). Estimation problems in the block-diagonal model of the multitrait-multimethod matrix. Applied Psychological Measurement, 14: 325-339.
12.
Browne, M.W. (1984). The decomposition of multitrait-multimethod matrices. British Journal of Mathematical and Statistical Psychology, 37: 1-21.
13.
Browne, M.W. (April, 1992). MUTMUM users guide. Department of Psychology, Ohio State University.
14.
Byrne, B.M. & Coffin, R.D. (1993). Modeling MTMM data from additive and multiplicative covariance structures: An audit of construct validity concordance. Multivariate Behavioral Research, 28: 67-96.
15.
Campbell, D.T. & Fiske, D.W. (1959). Convergent and discriminant validation by the multitrait-multimethod matrix. Psychological Bulletin, 56: 81-105.
16.
Campbell, D.T. & O’Connell, E.J. (1967). Method factors in multitrait-multimethod matrices: Multiplicative rather than additive?Multivariate Behavioral Research, 2: 409-426.
17.
Campbell, D. T. & O’Connell, E.J. (1982). Methods as diluting trait relationships rather than adding irrelevant systematic variance. Pp. 93-111 in D. Brinberg & L.H. Kidder (Eds.), New directions for methodology of social and behavioral science: Forms of validity in research. San Francisco, CA: Jossey-Bass.
18.
Cote, J.A. & Buckley, M.R. (1987). Estimating trait, method, and error variance: Generalizing across 70 construct validation studies. Journal of Marketing Research, 24: 315-318.
19.
Cudeck, R. (1989). Analysis of correlation matrices using covariance structure models. Psychological Bulletin, 105: 317-327.
20.
Dickinson, T.L., Hassett, C.E., & Tannenbaum, S.I. (1986, December). Work performance ratings: A meta-analysis of multitrait-multimethod studies (AFHRL-TP-86-32). Brooks Air Force Base. TX: Air Force Human Resources Laboratory.
21.
Dickinson, T.L. & Tice, T.E. (1973). A multitrait-multimethod analysis of scales developed by retranslation. Organizational Behavior and Human Performance, 9: 421-438.
22.
Dunham, R.B., Smith, F.J. & Blackburn, R.S. (1977). Validation of the index of organizational reactions with the JDI, the MSQ, and faces scales. Academy of Management Journal, 20: 420-432.
23.
Elbert, N.F. (1979). Questionnaire validation by confirmatory factor analysis: An improvement over multitrait-multimethod matrices. Decision Sciences, 10: 629-644.
24.
Forsythe, G.B., McGaghie, W.C. & Friedman, C.P. (1986). Construct validity of medical clinical competence measures: A multitrait-multimethod matrix study using confirmatory factor analysis. American Educational Research Journal, 23: 315-336.
25.
Freedman, R.D. & Stumpf, S.A. (1978). Student evaluations of courses and faculty based on a perceived learning criterion: Scale construction, validation, and comparison of results. Applied Psychological Measurement, 2: 189-202.
26.
Goffin, R.D. & Jackson, D.N. (1992). Analysis of multitrait-multirater performance appraisal data: Composite direct product method versus confirmatory factor analysis. Multivariate Behavioral Research, 27: 363-385.
Kavanagh, M.J., MacKinney, A.C. & Wolins, L. (1971). Issues in managerial performance: Multitrait-multimethod analyses of ratings. Psychological Bulletin, 75: 34-49.
33.
Kenny, D.A. (1976). An empirical application of confirmatory factor analyses to the multitrait-multimethod matrix. Journal of Experimental Social Psychology, 12: 247-252.
34.
Kenny, D.A. (1979). Correlation and causality. New York: Wiley.
35.
Kenny, D.A. (1994). The multitrait-multimethod matrix: Design, analysis, and conceptual issues. Pp. 111-124 in P.E. Shrout & ST. Fiske (Eds.), Personality, research, methods, and theory: A Festschrift honoring Donald W. Fiske. Hillsdale, NJ: Erlbaum.
36.
Kenny, D.A. & ­, D.A. (1992). Analysis of the multitrait-multimethod matrix by confirmatory factor analysis. Psychological Bulletin, 112: 165-172.
37.
King, L.M., Hunter, J.E. & Schmidt, F.L. (1980). Halo in a multidimensional forced-choice performance evaluation scale. Journal of Applied Psychology, 65: 507-516.
38.
Lance, C.E., Teachout, M.S. & Donnelly, T.M. (1992). Specification of the criterion construct space: An application of hierarchical confirmatory factor analysis, Journal of Applied Psychology, 77: 437-452.
39.
Lawler, E.E., III. (1967). The multitrait-multirater approach to measuring managerial job performance. Journal of Applied Psychology, 51: 369-381.
40.
Marsh, H.W. (1989). Confirmatory factor analyses of multitrait-multimethod data: Many problems and a few solutions. Applied Psychological Measurement, 13: 335-361.
41.
Marsh, H.W. & Bailey, M. (1991). Confirmatory factor analysis of multitrait-multimethod data: A comparison of alternative models. Applied Psychological Measurement, 15: 47-70.
42.
Marsh, H.W., Balla, J.R. & McDonald, R.P. (1988). Goodness-of-fit indexes in confirmatory factor analysis: The effect of sample size. Psychological Bulletin, 103: 391-410.
43.
Marsh, H.W. & Butler, S. (1984). Evaluating reading diagnostic tests: An application of confirmatory factor analysis to multitrait-multimethod data. Applied Psychological Measurement, 8: 307-320.
44.
McDonald, R.P. & Marsh, H.W. (1990). Choosing a multivariate model: Noncentrality and goodness of fit. Psychological Bulletin, 107: 247-255.
Orpen, C. (1973). An empirical assessment of the job performance of high-level executives by means of a multitrait-multimethod matrix. Psychologia Africana, 15: 7-14.
50.
Piotrowski, M.J. (1993). Ratings of human resource workers. Unpublished raw data.
51.
Pollak, F. (1994). Team assessments. Unpublished raw data.
52.
Rothstein, H.R. (1990). Interrater reliability of job performance ratings: Growth to asymptote level with increasing opportunity to observe. Journal of Applied Psychology, 75: 322-327.
53.
Schmitt, N., Coyle, B.W. & Saari, B.B. (1977). A review and critique of analyses of multitrait-multimethod matrices. Multivariate Behavioral Research, 12: 447-478.
54.
Schmitt, N. & Stults, D.M. (1986). Methodology review: Analysis of multitrait-multimethod matrices. Applied Psychological Measurement, 10: 1-22.
55.
Sypher, B.D. & Sypher, H.E. (1984). Seeing ourselves as others see us: Convergence and divergence in assessments of communication behavior. Communication Research, 12: 97-115
56.
Tanaka, J.S. (1987). “How big is big enough?“: Sample size and goodness of lit in structural equation models with latent variables. Child Development, 58: 134-146.
57.
Tucker, M.F., Cline, V.B. & Schmitt, J.R. (1967). Prediction of creativity and other performance measures from biographical information among pharmaceutical scientists. Journal of Applied Psychology, 51: 131-138.
Williams, L.J., Cote, J.A. & Buckley, M.R. (1989). Lack of method variance in self-reported affect and perceptions at work: Reality or artifact?Journal of Applied Psychology, 74: 462-468.
60.
Wothke, W. & Browne, M.W. (1990). The direct product model for the MTMM matrix parameterized as a second order factor analysis model. Psychometrika, 5.5: 255-262.
