This article presents a historical review of how inconsistency in assessment center ratings has been regarded among AC researchers and practitioners, then compares these perspectives to views of inconsistency found in personality psychology. Based on this review, the authors argue for a return to the study of consistency as an individual difference, rather than as simple measurement error. They offer four propositions regarding the inconsistency observed in AC performance, arguing that such inconsistency presents a unique opportunity to identify individuals’ patterns of skill proficiency. Finally, they discuss ways in which differences in consistency are likely to relate to organizational interests, including implications for selection and development.
Albrecht, P.A., Glaser, E.M., & Marks, J.1964. Validation of a multiple-assessment procedure for managerial personnel. Journal of Applied Psychology, 48: 351-360.
2.
Alker, H.A.1972. Is personality situationally specific or intrapsychically consistent? Journal of Personality, 40: 1-16.
3.
Allport, G.W.1937. Personality: A psychological interpretation. New York: Holt, Rinehart, & Winston.
4.
Archambeau, D.J.1979. Relationships among skill ratings assigned in an assessment center. Journal of Assessment Center Technology, 2: 7-19.
5.
Arthur, W., Jr., Day, E.A., McNelly, T.L., & Edens, P.S.2003. A meta-analysis of the criterion-related validity of assessment center dimensions. Personnel Psychology, 56: 125-154.
6.
Arthur, W., Jr., Woehr, D.J., & Maldegen, R.2000. Convergent and discriminant validity of assessment center dimensions: A conceptual and empirical reexamination of the assessment center construct-related validity paradox. Journal of Management , 26: 813-835.
7.
Ashton, M.C., Lee, K., Perugini, M., Szarota, P., de Vries, R.E., Di Blas, L., et al. 2004. A six-factor structure of personality-descriptive adjectives: Solutions from psycholexical studies in seven languages. Journal of Personality and Social Psychology, 86: 356-366.
8.
Beal, D.J., Weiss, H.M., Barros, E., & McDermid, S.M.2005. An episodic process model of affective influences on performance. Journal of Applied Psychology, 90: 1054-1068.
9.
Bem, D.J.1983. Further déjà vu in the search for cross-situational consistency: A response to Mischel and Peake. Psychological Review, 90: 390-393.
10.
Bem, D.J., & Allen, A.1974. On predicting some of the people some of the time: The search for cross-situational consistencies in behavior. Psychological Review, 81: 506-520.
11.
Block, J.1978. The Q-sort method in personality assessment and psychiatric research. Palo Alto, CA: Consulting Psychologists Press.
12.
Borman, W.C.1982. Validity of behavioral assessment for predicting military recruiter performance. Journal of Applied Psychology, 67: 3-9.
13.
Borman, W.C.1991. Job behavior, performance, and effectiveness. In M. D. Dunnette & L. M. Hough (Eds.), Handbook of industrial and organizational psychology (2nd ed., Vol. 2, pp. 271-326). Palo Alto, CA: Consulting Psychologists Press.
14.
Bray, D.W.1982. The assessment center and the study of lives. American Psychologist, 37: 180-189.
15.
Bray, D.W., & Campbell, R.J.1968. Selection of salesmen by means of an assessment center. Journal of Applied Psychology , 52: 36-41.
16.
Bray, D.W., Campbell, R.J., & Grant, D.L.1974. Formative years in business: A long-term AT&T study of managerial lives. New York: John Wiley.
17.
Bray, D.W., & Grant, D.L.1966. The assessment center in the measurement of potential for business management. Psychological Monographs: General and Applied, 80: 27.
18.
Brummel, B.J., Rupp, D.E., & Spain, S.M. In press. Constructing parallel simulation exercises for assessment centers and other forms of behavioral assessment. Personnel Psychology .
19.
Buss, A.H.1989. Personality as traits. American Psychologist , 44: 1378-1388.
20.
Bycio, P., Alvares, K.M., & Hahn, J.1987. Situational specificity in assessment center ratings: A confirmatory factor analysis. Journal of Applied Psychology , 72: 463-474.
21.
Byham, W.C.1970. Assessment centers for spotting future managers. Harvard Business Review, 48: 150-168.
22.
Campbell, D.T., & Fiske, D.W.1959. Convergent and discriminant validation by the multitrait-multimethod matrix. Psychological Bulletin, 56: 81-105.
23.
Campbell, J.P., Gasser, M.B., & Oswald, F.L.1996. The substantive nature of job performance variability . In K. R. Murphy (Ed.), Individual differences and behavior in organizations: 258-299. Hillsdale, NJ: Erlbaum.
24.
Caspi, A., Roberts, B.W., & Shiner, R.L. ( 2005). Personality development: Stability and change. Annual Review of Psychology, 56: 453-484.
25.
Chan, D.1996. Criterion and construct validation of an assessment centre . Journal of Occupational and Organizational Psychology, 69: 167-181.
26.
Deadrick, D.L., & Gardner, D.G.1997. Distributional ratings of performance levels and variability . Group and Organization Management, 22: 317-342.
27.
Epstein, S.1979. The stability of behavior: I. On predicting most of the people much of the time. Journal of Personality and Social Psychology, 37: 1097-1126.
28.
Epstein, S.1980. The stability of behavior: II. Implications for psychological research. American Psychologist, 35: 790-806.
29.
Epstein, S.1983. The stability of confusion: A reply to Mischel and Peake . Psychological Review, 90: 179-184.
30.
Epstein, S., & O’Brien, E.J.1985. The person-situation debate in historical and current perspective. Psychological Bulletin, 98: 513-537.
31.
Eysenck, M.W., & Eysenck, H.J.1980. Mischel and the concept of personality. British Journal of Psychology, 71: 191-204.
32.
Finkle, R.B.1976. Managerial assessment centers. In M. D. Dunnette (Ed.), Handbook of industrial and organizational psychology: 888-889. Chicago: Rand McNally.
33.
Fiske, D.W., Hanfmann, E., MacKinnon, D.W., Miller, J.G., & Murray, H.A.1996. Selection of personnel for clandestine operations: Assessment of men. Walnut Creek, CA: Aegean Park Press. (Originally published 1948)
34.
Fleeson, W.2001. Toward a structure- and process-integrated view of personality: Traits as density distributions of states. Journal of Personality and Social Psychology, 80: 1011-1027.
35.
Funder, D.C.1983. Three issues in predicting more of the people: A reply to Mischel and Peake. Psychological Review, 90: 283-289.
36.
Funder, D.C., & Colvin, C.1991. Explorations in behavioral consistency: Properties of persons, situations, and behaviors. Journal of Personality and Social Psychology, 60: 773-794.
37.
Furr, R.M., & Funder, D.C.2004. Situational similarity and behavioral consistency: Subjective, objective, variable-centered, and person-centered approaches. Journal of Research in Personality, 38: 421-447.
38.
Gaugler, B.B., Rosenthal, D.B., Thornton, G.C., & Bentson, C.1987. Meta-analysis of assessment center validity. Journal of Applied Psychology, 72: 493-511.
39.
Gaugler, B.B., & Thornton, G.C.1989. Number of assessment center dimensions as a determinant of assessor accuracy. Journal of Applied Psychology, 74: 611-618.
40.
Gibbons, A.M., & Rupp, D.E.2007, April. Inconsistency in assessment center performance: A meaningful individual difference?Paper presented at the 22nd annual meeting of the Society for Industrial and Organizational Psychology, New York.
41.
Gibbons, A.M., Rupp, D.E., Snyder, L.A., Holub, A.S., & Woo, S.E.2006. A preliminary investigation of developable dimensions . The Psychologist-Manager Journal, 9: 99-123.
42.
Haaland, S., & Christiansen, N.D.2002. Implications of trait-activation theory for evaluating the construct validity of assessment center ratings . Personnel Psychology, 55: 137-163.
43.
Harris, M.M., Becker, A.S., & Smith, D.E.1993. Does the assessment center scoring method affect the cross-situational consistency of ratings? Journal of Applied Psychology, 78: 675-678.
44.
Hartshorne, H., & May, M.A.1928. Studies in the nature of character: Vol. 1. Studies in deceit. New York: MacMillan.
45.
Highhouse, S., & Harris, M.M.1993. The measurement of assessment center situations: Bem’s template matching technique for examining exercise similarity. Journal of Applied Social Psychology, 23: 140-155.
46.
Hofmann, D.A., Jacobs, R., & Gerras, S.J.1992. Mapping individual performance over time. Journal of Applied Psychology, 77: 185-195.
47.
Howard, A.1974. An assessment of assessment centers. Academy of Management Journal, 17: 115-134.
48.
Howard, A.1997. A reassessment of assessment centers: Challenges for the 21st century. Journal of Social Behavior and Personality, 12: 13-52.
49.
Hulin, C.L., Henry, R.A., & Noon, S.L.1990. Adding a dimension: Time as a factor in the generalizability of predictive relationships. Psychological Bulletin, 107: 328-340.
50.
International Task Force on Assessment Center Guidelines.2000. Guidelines and ethical considerations for assessment center operations. Public Personnel Management, 29: 315-331.
51.
Jansen, P.G.W., & Stoop, B.A.M.2001. The dynamics of assessment center validity: Results of a 7-year study. Journal of Applied Psychology, 86: 741-753.
52.
Joyce, L.W., Thayer, P.W., & Pond, S.B.1994. Managerial functions: An alternative to traditional assessment center dimensions? Personnel Psychology, 47: 109-121.
53.
Kammrath, L.K., Mendoza-Denton, R., & Mischel, W.2005. Incorporating if . . . then . . . personality signatures in person perception: Beyond the person-situation dichotomy. Journal of Personality and Social Psychology, 88: 605-618.
54.
Kane, J.S.1986. Performance distribution assessment. In R. A. Berk (Ed.), Performance assessment: Methods and applications: 237-273. Baltimore: Johns Hopkins.
55.
Kolk, N.J., Born, M.P., & van der Flier, H.2002. Impact of common rater variance on construct validity of assessment center dimension judgments. Human Performance, 15: 325-338.
56.
Kolk, N.J., Born, M.P., & van der Flier, H.2003. The transparent assessment centre: The effects of revealing dimensions to candidates. Applied Psychology: An International Review, 52: 648-668.
57.
Krause, D.E., & Thornton, G.C. , III.2006, September. Survey of assessment center practices around the world. Paper presented at the 33rd International Congress on Assessment Center Methods , London.
58.
Kuptsch, C., Kleinmann, M., & Koller, O.1998. The chameleon effect in assessment centers: The influence of cross-situational behavioral consistency on the convergent validity of assessment centers. Journal of Social Behavior and Personality , 13: 102-116.
59.
Lance, C.E.2008. Why assessment centers don’t work the way they’re supposed to. Industrial and Organizational Psychology: Perspectives on Science and Practice, 1: 84-97.
60.
Lance, C.E., Foster, M.R., Gentry, W.A., & Thoresen, J.D.2004. Assessor cognitive processes in an operational assessment center. Journal of Applied Psychology , 89: 22-35.
61.
Lance, C.E., Lambert, T.A., Gewin, A.G., Lievens, F., & Conway, J.M.2004. Revised estimates of dimension and exercise variance components in assessment center postexercise dimension ratings. Journal of Applied Psychology, 89: 377-385.
62.
Lance, C.E., Newbolt, W.H., Gatewood, R.D., Foster, M.R., French, N.R., & Smith, D.E.2000. Assessment center exercise factors represent cross-situational specificity, not method bias. Human Performance, 13: 323-353.
63.
Lievens, F.1998. Factors which improve the construct validity of assessment centers: A review. International Journal of Selection and Assessment, 6: 141-152.
64.
Lievens, F.2001a. Assessor training strategies and their effects on accuracy, interrater reliability, and discriminant validity. Journal of Applied Psychology, 86: 255-264.
65.
Lievens, F.2001b. Assessors and use of assessment centre dimensions: A fresh look at a troubling issue. Journal of Organizational Behavior, 22: 203-221.
66.
Lievens, F.2002. Trying to understand the different pieces of the construct validity puzzle of assessment centers: An examination of assessor and assessee effects. Journal of Applied Psychology, 87: 675-686.
67.
Lievens, F., Chasteen, C.S., Day, E.A., & Christiansen, N.D.2004, April. Large-scale investigation of the role of trait activation theory for understanding assessment center convergent and discriminant validity. Paper presented at the 19th annual conference of the Society for Industrial and Organizational Psychology, Chicago.
68.
Lievens, F., & Conway, J.M.2001. Dimension and exercise variance in assessment center scores: A large-scale evaluation of multitrait-multimethod studies. Journal of Applied Psychology, 86: 1202-1222.
69.
MacKinnon, D.W.1974. How assessment centers were started in the United States . Pittsburgh, PA: Development Dimensions International.
70.
Magnusson, D.1976. The person and the situation in an interactional model of behavior. Scandinavian Journal of Psychology, 17: 253-271.
71.
Marcus, B., Goffin, R.D., Johnston, N.G., & Rothstein, M.G.2007. Personality and cognitive ability as predictors of typical and maximum managerial performance. Human Performance, 20: 275-285.
72.
McConnell, J.J., & Parker, T.C.1972. An assessment center program for multi-organizational use. Training and Development Journal, 26: 6-14.
73.
McCrae, R.R., Costa, P.T., Jr., Ostendorf, F., Angleitner, A., Hrebickova, M., Avia, M.D., et al. 2000. Nature over nurture: Temperament, personality, and life span development. Journal of Personality and Social Psychology, 78: 173-186.
74.
McNutt, K.1979. Behavioral consistency and assessment centers: A reconciliation of the literature. Journal of Assessment Center Technology , 2: 1-6.
75.
Mischel, W.1968. Personality and assessment. New York : John Wiley.
76.
Mischel, W.1984. Convergences and challenges in the search for consistency . American Psychologist, 39: 351-364.
77.
Mischel, W.2004. Toward an integrative science of the person. Annual Review of Psychology, 55: 1-22.
78.
Mischel, W., & Peake, P.K.1982. Beyond déjà vu in the search for cross-situational consistency. Psychological Review, 89: 730-755.
79.
Mischel, W., Shoda, Y., & Mendoza-Denton, R. (2002). Situation-behavior profiles as a locus of consistency in personality. Current Directions in Psychological Science, 11: 50-54.
80.
Neidig, R.D., Martin, J.C., & Yates, R.E.1979. The contribution of exercise skill ratings to final assessment center evaluations. Journal of Assessment Center Technology , 2: 21-23.
81.
Newcomb, T.M.1929. The consistency of certain introvert-extrovert behavior patterns in 51 problem boys. New York: Columbia University, Teachers College.
82.
Pervin, L.A.1976. A free-response description approach to the analysis of person-situation interaction. Journal of Personality and Social Psychology, 34: 465-474.
83.
Rabbitt, P., Osman, P., Moore, B., & Stollery, B.2001. There are stable individual differences in performance variability, both from moment to moment and day to day. Quarterly Journal of Experimental Psychology, 54: 981-1003.
84.
Reb, J., & Cropanzano, R.2007. Evaluating dynamic performance: The influence of salient gestalt characteristics on performance ratings. Journal of Applied Psychology, 92: 490-499.
85.
Reilly, R.R., Henry, S., & Smither, J.W.1990. An examination of the effects of using behavior checklists on the construct validity of assessment center dimensions. Personnel Psychology, 43: 71-84.
86.
Robie, C., Osburn, H.G., Morris, M.A., Etchegaray, J.M., & Adams, K.A.2000. Effects of the rating process on the construct validity of assessment center dimension evaluations. Human Performance , 13: 355-370.
87.
Rupp, D.E., Thornton, G.C., III , & Gibbons, A.M.2008. The construct validity of the assessment center method and usefulness of dimensions as focal constructs . Industrial and Organizational Psychology: Perspectives on Science and Practice, 1: 116-120.
88.
Russell, C.J., & Domm, D.R.1991, April. Simultaneous ratings of role congruency and person characteristics, skills, and abilities in an assessment center: A field test of competing explanations. Paper presented at the sixth annual meeting of the Society for Industrial and Organizational Psychology , St. Louis, MO.
89.
Sackett, P.R., & Dreher, G.F.1982. Constructs and assessment center dimensions: Some troubling empirical findings. Journal of Applied Psychology, 67: 401-410.
90.
Sackett, P.R., & Dreher, G.F.1984. Situation specificity of behavior and assessment center validation strategies: A rejoinder to Neidig and Neidig. Journal of Applied Psychology, 69: 187-190.
91.
Sackett, P.R., & Tuzinski, K.A.2001. The role of dimensions and exercises in assessment center judgments. In M. London (Ed.), How people evaluate others in organizations. Applied in psychology: 111-129. Mahwah, NJ: Lawrence Erlbaum.
92.
Salthouse, T.A.2007. Implications of within-person variability in cognitive and neuropsychological functioning for the interpretation of change. Neuropsychology, 21: 401-411.
93.
Schleicher, D.J., Day, D.V., Mayes, B.T., & Riggio, R.E.2002. A new frame for frame-of-reference training: Enhancing the construct validity of assessment centers. Journal of Applied Psychology, 87: 735-746.
94.
Schneider, J.R., & Schmitt, N.1992. An exercise design approach to understanding assessment center dimension and exercise constructs. Journal of Applied Psychology, 77: 32-41.
95.
Shoda, Y.1999. A unified framework for the study of behavioral consistency: Bridging Person × Situation interaction and the consistency paradox . European Journal of Personality, 13: 361-387.
96.
Shoda, Y., Mischel, W., & Wright, J.C.1994. Intraindividual stability in the organization and patterning of behavior: Incorporating psychological situations into the idiographic analysis of personality. Journal of Personality and Social Psychology , 67: 674-687.
97.
Snyder, M.1974. Self-monitoring of expressive behavior. Journal of Personality and Social Psychology, 30: 526-537.
98.
Spychalski, A.C., Quinones, M.A., Gaugler, B.B., & Pohley, K.1997. A survey of assessment center practices in organizations in the United States. Personnel Psychology, 50: 71-90.
99.
Steiner, D.D., Rain, J.S., & Smalley, M.M.1993. Distributional ratings of performance: Further examination of a new rating format. Journal of Applied Psychology, 78: 438-442.
100.
Thornton, G.C., III , & Byham, W.C.1982. Assessment centers and managerial performance. New York: Academic Press.
101.
Thornton, G.C., III , & Mueller-Hanson, R.2003. Developing organizational simulations: A how-to guide for practitioners. Mahwah, NJ: Lawrence Erlbaum.
102.
Thornton, G.C., III , & Rupp, D.E.2005. Assessment centers in human resource management. Mahwah, NJ: Lawrence Erlbaum.
103.
Thornton, G.C., III , & Zorich, S.1980. Training to improve observer accuracy. Journal of Applied Psychology, 65: 351-354.
104.
Trankell, A.1959. The psychologist as an instrument of prediction. Journal of Applied Psychology, 43: 170-175.
105.
Turnage, J.J., & Muchinsky, P.M.1982. Transsituational variability in human performance within assessment centers. Organizational Behavior and Human Performance, 30: 174-200.
106.
Turner, R.G., & Gilliam, B.J.1979. Identifying the situationally variable subject: Correspondence among different self-report formats. Applied Psychological Measurement, 3: 361-369.
107.
Tziner, A., Ronen, S., & Hacohen, D.1993. A four-year validation study of an assessment center in a financial corporation. Journal of Organizational Behavior , 14: 225-237.
108.
Yang, Y., Read, S.J., & Miller, L.C.2006. A taxonomy of situations from Chinese idioms. Journal of Research in Personality, 40: 750-778.
109.
Zuckerman, M., Bernieri, F., Koestner, R., & Rosenthal, R.1989. To predict some of the people some of the time: In search of moderators. Journal of Personality and Social Psychology , 57: 279-293.
110.
Zuckerman, M., Koestner, R., DeBoy, T., Garcia, T., Maresca, B.C., & Sartoris, J.M.1988. To predict some of the people some of the time: A reexamination of the moderator variable approach in personality theory. Journal of Personality and Social Psychology, 54: 1006-1019.
