Three major reviews of the assessment center (AC) construct-validity literature have disagreed as to the most appropriate analytic model for AC postexercise dimension ratings. We report a Monte Carlo study addressing the following questions: (a) To what extent does the “true” model (i.e., the model that generated the data) actually appear to fit the data well? (b) To what extent can a model appear to fit the data well even though it is the wrong model? and (c) Is model fit actually a useful empirical criterion for judging which model is most likely the population model? Results suggest that “true” models may not always appear as the best fitting models, whereas “false” models sometimes appear to offer better fit than the true models. Implications for the study of AC construct validity are discussed.
Arthur, W.A., Jr., Woehr, D.J., & Maldegen, R. (2000). Convergent and discriminant validity of assessment center dimensions: A conceptual and empirical re-examination of the assessment center construct-related validity paradox. Journal of Management , 26, 813—835.
2.
Bagozzi, R.P., & Yi, Y. (1990). Assessing method variance in multitrait-multimethod matrices: The case of self-reported affect and perceptions at work. Journal of Applied Psychology, 75, 547—560.
3.
Bentler, P.M. (1990). Comparative fit indexes in structural models. Psychological Bulletin,107, 238—246.
4.
Bowler, M. & Woehr, D.J. (in press). A meta-analytic evaluation of the impact of dimension and exercise factors on assessment center ratings. Journal of Applied Psychology.
5.
Brannick, M.T., & Spector, P.E. (1990). Estimation problems in the block-diagonal model of the multitrait-multimethod matrix. Applied Psychological Measurement, 14, 325—339.
6.
Browne, M.W. (1984). The decomposition of multitrait-multimethod matrices . British Journal of Mathematical and Statistical Psychology , 33, 184—199.
7.
Buffett, J. (1999). I don't know and I don't care. On Beach house on the moon [CD]. Key West, FL: Margaritaville.
8.
Campbell, D.T., & Fiske, D.W. (1959). Convergent and discriminant validation by the multitrait-multimethod matrix. Psychological Bulletin,56, 81—105.
9.
Conway, J.M., Lievens, F., Scullen, S.E., & Lance, C.E. (2004). Bias in the correlated uniqueness model for MTMM data: A simulation study. Structural Equation Modeling, 11, 535—559.
10.
Eid, M. ( 2000). A multitrait-multimethod model with minimal assumptions . Psychometrika, 65, 241—261.
11.
Eid, M., Lischetzke, T., Nussbeck, F.W., & Trierweiler, L.I. (2003). Separating trait effects from trait-specific method effects in multitrait-multimethod models: A multiple- indicator CT-C(M-1) model. Psychological Methods, 8, 38—60.
12.
Hu, L.T., & Bentler, P.M. (1998). Fit indices in covariance structure modeling: Sensitivity to underparameterization model misspecification. Psychological Methods, 3, 424—453.
13.
Hu, L.T., & Bentler, P.M. (1999). Cutoff criteria for fit indexes in covariance structure analysis. Structural Equation Modeling, 6, 1—55.
Kenny, D.A., & Kashy, D.A. (1992). Analysis of the multitrait-multimethod matrix by confirmatory factor analysis. Psychological Bulletin,112, 165—172.
16.
Lance, C.E., Foster, M.R., Gentry, W.A., & Thoresen, J.D. (2004). Assessor cognitive processes in an operational assessment center. Journal of Applied Psychology , 89, 22—35.
17.
Lance, C.E., Lambert, T.A., Gewin, A.G., Lievens, F., & Conway, J.M. (2004). Revised estimates of dimension and exercise variance components in assessment center postexercise dimension ratings. Journal of Applied Psychology, 89, 377—385.
18.
Lance, C.E., Newbolt, W.H., Gatewood, R.D., Foster, M.R., French, N., & Smith, D.E. (2000). Assessment center exercise factors represent cross-situational specificity, not method bias. Human Performance, 13, 323—353.
19.
Lance, C.E., Noble, C.L., & Scullen, S.E. (2002). A critique of the correlated trait-corrrelated method and correlated uniqueness models for multitrait-multimethod data. Psychological Methods, 7, 228—244.
20.
Lievens, F., & Conway, J.M. (2001). Dimension and exercise variance in assessment center scores: A large-scale evaluation of multitrait-multimethod studies . Journal of Applied Psychology, 86, 1202—1222.
21.
Lievens, F., & Klimoski, R.J. (2001). Understanding the assessment centre process: Where are we now?International Review of Industrial and Organizational Psychology, 16, 246—286.
22.
Marsh, H.W. (1989). Confirmatory factor analysis of multitrait-multimethod data: Many problems and a few solutions. Applied Psychological Measurement, 13, 335—361.
23.
Marsh, H.W., & Bailey, M. (1991). Confirmatory factor analysis of multitrait-multimethod data: A comparison of alternative models. Applied Psychological Measurement, 15, 47—70.
24.
Marsh, H.W., Byrne, B.M., & Craven, R. (1992). Overcoming problems in confirmatory factor analyses of MTMM data: The correlated uniqueness model and factorial invariance. Multivariate Behavioral Research, 27, 489—507.
25.
Marsh, H.W., & Hocevar, D. (1988). A new, more powerful approach to multitrait-multimethod analyses: Application of second-order confirmatory factor analyses. Journal of Applied Psychology, 73, 107—117.
26.
Quartetti, D.A., & Mulaik, S.A. (1997). First order or higher order general factor?Structural Equation Modeling, 4, 193—211.
27.
Sackett, P.R., & Tuzinski, K. (2001). The role of dimensions and exercises in assessment center judgments. In M. London (Ed.), How people evaluate others in organizations (pp. 111—129). Mahwah, NJ: Lawrence Erlbaum.
28.
Schweizer, K. (2003). The role of expectations in investigating multitrait-multimethod matrices by Procrustes rotation. Methods of Psychological Research, 1, 1—19.
29.
Scullen, S.E. (1999). Using confirmatory factor analysis of correlated uniquenesses to estimate method variance in multitrait-multimethod matrices . Organizational Research Methods, 2, 275—292.
Widaman, K.F. (1992). Miltitrait-multimethod models in aging research . Experimental Aging Research, 18, 185—201.
32.
Woehr, D.J., and Arthur, W., Jr., (2003). The construct-related validity of assessment center ratings: A review and meta-analysis of the role of methodological factors . Journal of Management, 29, 231—258.
33.
Wothke, W., & Browne, M.W. (1990). The direct product model for the MTMM matrix parameterized as a second order factor analysis model. Psychometrika, 55, 255—262.
