The authors illustrate a problem with confirmatory factor analysis (CFA)-based strategies to model disaggregated multitrait—multirater (MTMR) data—the potential to find markedly different results with the same sample of ratees simply as a result of how one selects and identifies raters within the data set one has gathered for analysis. Using performance ratings gathered as part of a large criterion-related validation study, the authors show how such differences manifest themselves in several ways including variation in (a) covariance matrices that serve as input for the modeling effort, (b) model convergence, (c) admissibility of solutions, (d) overall model fit, (e) model parameter estimates, and (f) model selection. Implications of this study for past research and recommendations for future CFA-based MTMR modeling efforts are discussed.
Balzer, W.K., Greguras, G.J., & Raymark, P.H. ( 2004). Multisource feedback. In J. C. Thomas (Ed.), Comprehensive handbook of psychological assessment, Vol. 4: Industrial and organizational assessment (pp. 390-411). Hoboken, NJ: John Wiley.
2.
Barr, M.A., & Raju, N.S. ( 2003). IRT-based assessments of rater effects in multiple-source feedback instruments. Organizational Research Methods, 6, 15-43.
3.
Bentler, P.M., & Bonett, D.G. ( 1980). Significance tests and goodness of fit in the analysis of covariance structures. Psychological Bulletin, 88, 588-606.
4.
Borman, W.C., & Motowidlo, S.J. (1997). Task performance and contextual performance: The meaning for personnel selection research. Human Performance, 10, 99-109.
5.
Bracken, D. W., Timmreck, C. W., & Church, A. H. (Eds.). (2001). The handbook of multisource feedback: The comprehensive resource for designing and implementing MSF processes. San Francisco: Jossey-Bass.
6.
Brannick, M.T., & Spector, P.E. ( 1990). Estimation problems in the block-diagonal model of the multitrait-multimethod matrix. Applied Psychological Measurement , 14, 325-339.
7.
Burnham, K.P., & Anderson, D.R. (2002). Model selection and multimodel inference: A practical information-theoretic approach (2nd ed.). New York: Springer.
8.
Campbell, J. P., & Knapp, D. J. (Eds.). (2001). Exploring the limits in personnel selection and classification. Mahwah, NJ: Lawrence Erlbaum.
9.
Conway, J.M. ( 1996). Analysis and design of multitrait-multirater performance appraisal studies. Journal of Management, 22, 139-162.
10.
Conway, J.M., & Huffcutt, A.I. (1997). Psychometric properties of multisource performance ratings: A meta-analysis of subordinate, supervisor, peer, and self-ratings. Human Performance, 10, 331-360.
11.
Craig, S.B., & Kaiser, R.B. ( 2003). Applying item response theory to multisource performance ratings: What are the consequences of violating the independent observations assumption? Organizational Research Methods, 6, 44-60.
12.
Cronbach, L.J., Gleser, G.C., Nanda, H., & Rajaratnam, N. (1972). The dependability of behavioral measurements: Theory of generalizability for scores and profiles. New York: John Wiley.
13.
Eid, M., Nussbeck, F.W., Geiser, C., Gollwitzer, M., Cole, D.A., & Lischetzke, T.A. (2008). Structural equation modeling of multitrait-multimethod data: Different models for different types of methods . Psychological Methods, 13, 230-253.
14.
Ewen, A.J., & Edwards, M.R. ( 2001). Readiness for multisource feedback. In D. W. Bracken, C. W. Timmreck , & A. H. Church (Eds.), The handbook of multisource feedback (pp. 33-47). San Francisco : Jossey-Bass.
15.
Greguras, G.J., Robie, C., Schleicher, D.J., & Goff, M. III. (2003). A field study of the effects of rating purpose on the quality of multisource ratings. Personnel Psychology , 56, 1-21.
16.
Harris, M.M., Becker, A.S., & Smith, D.E. ( 1993). Does the assessment center scoring method affect the cross-situational consistency of ratings? Journal of Applied Psychology, 78, 675-678.
17.
Hoffman, B.J., Lance, C.E., Bynum, B.H., & Gentry, W.A. ( 2010). Rater source effects are alive and well after all. Personnel Psychology, 63, 119-151.
18.
Hoyt, W.T., & Kerns, M. ( 1999). Magnitude and moderators of bias in observer ratings: A meta-analysis. Psychological Methods, 4, 403-424.
19.
Hu, L.T., & Bentler, P.M. ( 1999). Cutoff criteria for fit indexes in covariance structure analysis. Structural Equation Modeling, 6, 1-55.
20.
Hunter, J.E., & Schmidt, F.L. ( 2004). Methods of meta-analysis: Correcting error and bias in research findings (2nd ed.). Thousand Oaks, CA: SAGE.
21.
Kenny, D.A., & Kashy, D.A. ( 1992). Analysis of multitrait-multimethod matrix by confirmatory factor analysis. Psychological Bulletin, 112, 165-172.
22.
Knapp, D.J., Sager, C.E., & Tremble, T.R. ( 2005). Development of experimental Army enlisted personnel selection and classification tests and job performance criteria (Technical Report1168). Arlington, VA : United States Army Research Institute for the Behavioral and Social Sciences.
23.
LaHuis, D.M., & Avis, J.M. ( 2007). Using multilevel random coefficient modeling to investigate rater effects in performance ratings. Organizational Research Methods, 10, 97-107.
24.
Lance, C.E., Noble, C.L., & Scullen, S.E. ( 2002). A critique of the correlated trait-Correlated method (CTCM) and correlated uniqueness (CU) models for multitrait-multimethod (MTMM) data. Psychological Methods, 7, 228-244.
25.
Luo, W., & Kwok, O. ( 2009). The impacts of ignoring a crossed factor in analyzing cross-classified data. Multivariate Behavioral Research, 44, 182-212.
26.
Marsh, H.W., & Grayson, D. ( 1995). Latent variable models of multitrait-multimethod data . In R. H. Hoyle (Ed.), Structural equation modeling (pp. 177-198). Thousand Oaks, CA: SAGE.
Meyers, J.L., & Beretvas, S.N. (2006). The impact of inappropriate modeling of cross-classified data structures. Multivariate Behavioral Research, 41, 473-497.
Pulakos, E.D., & Schmitt, N. ( 1995). Experience-based and situational interview questions: Studies of validity. Personnel Psychology, 48, 289-308.
33.
Putka, D.J., Ingerick, M., & McCloy, R.A. (2008). Integrating traditional perspectives on error in ratings: Capitalizing on advances in mixed effects modeling. Industrial and Organizational Psychology, 1, 167-173.
34.
Putka, D.J., Le, H., McCloy, R.A., & Diaz, T. ( 2008). Ill-structured measurement designs in organizational research: Implications for estimating interrater reliability. Journal of Applied Psychology, 93, 959-981.
35.
Rasbash, J., & Browne, W.J. ( 2008). Non-hierarchical multilevel models. In J. de Leeuw & E. Meijer (Eds.), Handbook of multilevel analysis (pp. 301-334). New York: Springer.
36.
Rasbash, J., & Goldstein, H. ( 1994). Efficient analysis of mixed hierarchical and cross-classified random structures using a multilevel model. Journal of Educational and Behavioral Statistics, 19, 337-350.
37.
Scullen, S.E., Mount, M.K., & Goff, M. ( 2000). Understanding the latent structure of job performance ratings. Journal of Applied Psychology, 85, 956-970.
38.
Scullen, S.E., Mount, M.K., & Judge, T.A. ( 2003). Evidence of the construct validity of developmental ratings of managerial performance. Journal of Applied Psychology, 88, 50-66.
39.
Van Iddekinge, C.H., Raymark, P.H., Eidson, C.E., & Attenweiler, W.J. (2004). What do structured selection interviews really measure? The construct validity of behavior description interviews. Human Performance, 17, 71-93.
40.
Viswesvaran, C., Ones, D.S., & Schmidt, F.L. ( 1996). Comparative analysis of the reliability of job performance ratings. Journal of Applied Psychology, 81, 557-574.
41.
Viswesvaran, C., Schmidt, F.L., & Ones, D.S. ( 2005). Is there a general factor in ratings of job performance? A meta-analytic framework for disentangling substantive and error influences . Journal of Applied Psychology, 90, 108-131.
42.
Woehr, D.J., Sheehan, M.K., & Bennett, W. Jr. (2005). Assessing measurement equivalence across rating sources: A multitrait-multimethod approach. Journal of Applied Psychology, 90, 592-600.
