Convergent and Discriminant Validity of Assessment Center Dimensions: A Conceptual and Empirical Reexamination of the Assessment Center Construct-Related Validity Paradox

Abstract

This study notes that the lack of convergent and discriminant validity of assessment center ratings in the presence of content-related and criterion-related validity is paradoxical within a unitarian framework of validity. It also empirically demonstrates an application of generalizability theory to examining the convergent and discriminant validity of assessment center dimensional ratings. Generalizability analyses indicated that person, dimension, and person by dimension effects contribute large proportions of variance to the total variance in assessment center ratings. Alternately, exercise, rater, person by exercise, and dimension by exercise effects are shown to contribute little to the total variance. Correlational and confirmatory factor analyses results were consistent with the generalizability results. This provides strong evidence for the convergent and discriminant validity of the assessment center dimension ratings–a finding consistent with the conceptual underpinnings of the unitarian view of validity and inconsistent with previously reported results. Implications for future research and practice are discussed.

Get full access to this article

View all access options for this article.

References

Arthur, W., Jr , & Benjamin, L. T., Jr. 1999. Psychology applied to business. In A. M. Stec , & D. A. Bernstein (Eds.), Psychology: Fields of application: 98-115. Boston, MA: Houghton Mifflin.

Arthur, W., Jr , & Tubre, T. C. 1999. The assessment center construct-related validity paradox: A case of construct misspecification? In Quiñones, M. A. (Chair), Assessment centers, 21st century: New issues and new answers to old problems. Symposium presented at the 14th Annual Conference of the Society for Industrial and Organizational Psychology, Atlanta, GA.

Bender, J. M. 1973. What is “typical” of assessment centers?. Personnel, 50: 50–57.

Bernardin, H. J. , & Buckley, M. R. 1981. Strategies in rater training. Academy of Management Review, 6: 205–212.

Binning, J. F. , & Barrett, G. V. 1989. Validity of personnel decisions: A conceptual analysis of the inferential and evidential bases. Journal of Applied Psychology, 74: 478–494.

Brannick, M. T. , Michaels, C. E. , & Baker, D. P. 1989. Construct validity of in-basket scores. Journal of Applied Psychology, 74: 957–963.

Brennan, R. L. 1994. Variance components in generalizability theory. In C. R. Reynolds (Ed.), Cognitive assessment: A multidisciplinary perspective: 175–207. NY: Plenum Press.

Bycio, P. , Alvares, K. M. , & Hahn, J. 1987. Situation specificity in assessment center ratings: A confirmatory analysis. Journal of Applied Psychology, 72: 463–474.

Church, A. H. 1997. Managerial self-awareness in high-performing individuals in organizations. Journal of Applied Psychology, 82: 281–292.

10.

Cronbach, L. J. , Glesser, G. C. , Nanda, H. , & Rajaratnam, N. 1972. The dependability of behavioral measurements: Theory of generalizability for scores and profiles. New York: John Wiley & Sons, Inc.

11.

Cronshaw, S. F. , & Ellis, R. J. 1991. A process investigation of self-monitoring and leader emergence. Small Group Research, 22: 403–420.

12.

Dugan, B. 1988. Effects of assessor training on information use. Journal of Applied Psychology, 73: 743–748.

13.

Gaugler, B. B. , Rosenthal, D. B. , Thornton, G. C. III , & Bentson, B. 1987. Meta-analysis of assessment center validity. Journal of Applied Psychology, 72: 493–511.

14.

Gaugler, B. B. , & Thornton, G. C., III . 1989. Number of assessment center dimensions as a determinant of assessor generalizability of the assessment center ratings. Journal of Applied Psychology, 74: 611–618.

15.

Harris, M. M. , Becker, A. S. , & Smith, D. E. 1993. Does the assessment center scoring method affect the cross-situational consistency of ratings?. Journal of Applied Psychology, 78: 675–678.

16.

Highhouse, S. , & Harris, M. M. 1993. The measurement of assessment center situations: Bem’s template matching technique for examining exercise similarity. Journal of Applied Social Psychology, 23: 140–155.

17.

Howard, A. 1997. A reassessment of assessment centers: Challenges for the 21st century. Journal of Social Behavior and Personality, 12: 13–52.

18.

Jones, R. G. 1992. Construct validation of assessment center final dimension ratings: Definition and measurement issues. Human Resources Management Review, 2: 195–220.

19.

Jones, R. G. , & Whitmore, M. D. 1995. Evaluating developmental assessment centers as interventions. Personnel Psychology, 48: 377–388.

20.

Klimoski, R. , & Brickner, M. 1987. Why do assessment centers work? The puzzle of assessment center validity. Personnel Psychology, 40: 243–259.

21.

Kraiger, K. , & Teachout, M. S. 1990. Generalizability theory as construct-related evidence of the validity of job performance ratings. Human Performance, 3: 19–35.

22.

Lance, C. E., Gatewood, R. D., Newbolt, W. H., Foster, M. S., French, N. R., & Smith, D. E. 1998. Assessment center exercise factors represent cross-situational specificity, not method bias. Unpublished manuscript. University of Georgia, Athens, GA.

23.

Landy, F. J. 1986. Stamp collecting versus science: Validation as hypothesis testing. American Psychologist, 41: 1181–1192.

24.

Lievens, F. 1998. Factors which improve the construct validity of assessment centers: A review. International Journal of Selection and Assessment, 6: 141–152.

25.

Marsh, H. W. , & Grayson, D. 1995. Latent variable models of multitrait-multimethod data. In R. Hoyle (Ed.), Structural equation modeling: Concepts, issues, and applications. Thousand Oaks, CA: Sage Publications.

26.

McHenry, K. R. , & Schmitt, N. 1994. Multimedia testing. In M. J. Rumsey , C. D. Walker , & J. Harris (Eds.), Personnel selection and classification research: 193–222. Mahwah, NJ: Lawrence Erlbaum.

27.

Metropolitan Transit Authority. 1972. The uses of the assessment center in a government agency’s management program. Unpublished manuscript.

28.

Miller, G. A. 1956. The magical number seven, plus or minus two: Some limits on our capacity for processing information. Psychological Review, 63: 81–97.

29.

Neidig, R. D. , & Neidig, P. J. 1984. Multiple assessment center exercises and job relatedness. Journal of Applied Psychology, 69: 182–186.

30.

Nunnally, J. C. , & Bernstein, I. H. 1994. Psychometric theory (3rd ed.). New York: McGraw-Hill.

31.

Raymark, P. H. , & Binning, J. F. 1997. Explaining assessment center validity: A test of the criterion contamination hypothesis. Paper presented at the 1997 Academy of Management meeting, Boston, MA.

32.

Ritchie, R. J. , & Moses, J. L. 1983. Assessment center correlates of women’s advancement into middle management: A 7-year longitudinal analysis. Journal of Applied Psychology, 68: 227–231.

33.

Rowe, F. A. , & Mauer, K. A. 1991. Career guidance, career assessment, and consultancy. Journal of Career Development, 17: 223–233.

34.

Russell, C. 1985. Individual decision processes in an assessment center. Journal of Applied Psychology, 70: 737–746.

35.

Russell, C. J. , & Domm, D. R. 1995. Two field test of an explanation of assessment centre validity. Journal of Occupational and Organizational Psychology, 68: 25–47.

36.

Saal, F. E. , Downey, R. G. , & Lahey, M. A. 1980. Rating the ratings: Assessing the psychometric quality of rating data. Psychological Bulletin, 88: 413–428.

37.

Sackett, P.R. 1987. Assessment centers and content validity: Some neglected issues. Personnel Psychology, 40: 13–25.

38.

Sackett, P. R. , & Dreher, G. F. 1982. Constructs and assessment center dimensions: Some troubling empirical findings. Journal of Applied Psychology, 67: 401–410.

39.

Sackett, P. R. , & Hakel, M. D. 1979. Temporal stability and individual differences in using assessment center information from overall ratings. Organizational Behavior and Human Performance, 23: 120–137.

40.

Sackett, P. R. , & Harris, M. M. 1988. A further examination of the constructs underlying assessment center ratings. Journal of Business and Psychology, 3: 214–229.

41.

Sagie, A. , & Magnezy, R. 1997. Assessor type, number of distinguishable categories, and assessment centre construct validity. Journal of Occupational and Organizational Psychology, 67: 401–410.

42.

SAS Institute . 1990. SAS/STAT user’s guide (version 6, 4th ed., vol. 2). Cary, NC: SAS Institute Inc.

43.

Schmitt, N. 1977. Interrater agreement in dimensionality and combination of assessment center judgments. Journal of Applied Psychology, 62: 171–176.

44.

Schmitt, N. , Schneider, J. R. , & Cohen, S. A. 1990. Factors affecting validity of a regionally administered assessment center. Personnel Psychology, 43: 1–12.

45.

Schneider, J. R. , & Schmitt, N. 1992. An exercise design approaach to understanding assessment center dimension and exercise constructs. Journal of Applied Psychology, 77: 32–41.

46.

Shavelson, R. J. , & Webb, N. M. 1991. Generalizability theory: A primer, vol 1. Newbury Park, CA: Sage Publications.

47.

Silverman, W. H. , Dalessio, A. , Woods, S. B. , & Johnson, R. L., Jr. 1986. Influence of assessment center methods on assessors’ ratings. Personnel Psychology, 39: 565–578.

48.

Spychalski, A. C. , Quiñones, M. A. , Gaugler, B. B. , & Pohley, K. 1997. A survey of assessment center practices in organizations in the United States. Personnel Psychology, 50: 71–90.

49.

Task Force on Assessment Center Guidelines . 1989. Guidelines for ethical considerations. Public Personnel Management, 18: 457–470.

50.

Thornton, G. C. III , & Byham, W. C. 1982. Assessment centers and managerial performance. NY: Academic Press.

51.

Turnage, J. J. , & Muchinsky, P. M. 1982. Transsituational variability in human performance within assessment centers. Organizational Behavior and Human Performance, 30: 174–200.

52.

Woehr, D. J. , & Arthur, W., Jr. 1999. The assessment center validity paradox: A review of the role of methodological factors. In Quiñones, M. A. (Chair), Assessment centers, 21st century: New issues and new answers to old problems. Symposium presented at the 14th Annual Conference of the Society for Industrial and Organizational Psychology, Atlanta, GA.

53.

Woehr, D. J. , & Huffcutt, A. I. 1994. Rater training for performance appraisal: A meta-analytic review. Journal of Occupational and Organizational Psychology, 67: 189–205.