Multisource performance ratings involve multiple raters of a common target, thus violating the statistical assumption of independent observations. This study investigated the consequences of that violation on item response theory–based analyses of ratings in two domains of managerial performance. Using a differential item functioning framework, the authors found no significant differences in item parameters attributable to the violation. This finding suggests that item response theory can be applied to multisource ratings without discarding data in order to meet the independence assumption.
Get full access to this article
View all access options for this article.
References
1.
Atwater, L., & Yammarino, F. J. (1997). Antecedents and consequences of self-other rating agreement: A review and model. In G. Ferris (Ed.), Research in personnel and human resources management. Greenwich, CT: JAI.
2.
Baker, F. (1995). EQUATE 2.1: Computer program for equating two metrics in item response theory. Madison: University of Wisconsin, Laboratory of Experimental Design.
3.
Bass, B. M. (1990). Bass and Stogdill’s handbook of leadership: Theory, research, and managerial applications (3rd ed.). New York: Free Press.
4.
Bock, R. D., & Aitkin, M. (1981). Marginal maximum likelihood estimation of item parameters: An application of an EM algorithm. Psychometrika, 46, 443-459.
5.
Campion, M. A., Pursell, E. D., & Brown, B. K. (1988). Structured interviewing: Raising the psychometric properties of the employment interview. Personnel Psychology, 41, 25-42.
6.
Cattell, R. B. (1966). The meaning and strategic use of factor analysis. In R. B. Cattell (Ed.), Handbook of multivariate experimental psychology (pp. 174-243). Chicago: Rand McNally.
7.
Church, A. H. (1997). Managerial self-awareness in high-performing individuals in organizations. Journal of Applied Psychology, 82, 281-292.
8.
Conway, J. M., & Huffcutt, A. I. (1997). Psychometric properties of mulitsource performance ratings: A meta-analysis of subordinate, supervisor, peer, and self-ratings. Human Performance, 10, 331-360.
9.
Conway, J. M., Jako, R. A., & Goodman, D. F. (1995). A meta-analysis of interrater and internal consistency reliability of selection interviews. Journal of Applied Psychology, 80, 565-579.
10.
Facteau, J., & Craig, S. B. (2001). Are performance appraisal ratings obtained from different rating sources comparable? Journal of Applied Psychology, 86, 215-227.
11.
Fiedler, F. E. (1967). A theory of leadership effectiveness. New York: McGraw-Hill.
12.
Fleenor, J. W., McCauley, C. D., & Brutus, S. (1996). Self-other rating agreement and leader effectiveness. Leadership Quarterly, 7, 487-506.
13.
Fleishman, E. A. (1973). Twenty years of consideration and structure. In E. A. Fleishman & J. G. Hunt (Eds.), Current developments in the study of leadership. Carbondale: Southern Illinois University Press.
14.
Greguras, G. J., & Robie, C. (1998). A new look at within-source interrater reliability of 360-degree feedback ratings. Journal of Applied Psychology, 83, 960-968.
15.
Harris, M. M., Becker, A. S., & Smith, D. E. (1993). Does the assessment center scoring method affect the cross-situational consistency of ratings? Journal of Applied Psychology, 78, 675-678.
16.
Hazucha, J. F., Hezlett, S. A., & Schneider, R. J. (1993). The impact of 360-degree feedback on management skills development. Human Resource Management, 32, 325-351.
17.
Kaiser, H. F. (1960). The application of electronic computers to factor analysis. Educational and Psychological Measurement, 20, 141-151.
18.
Lombardo, M. M., & McCauley, C. D. (1994). Benchmarks: A manual and trainer’s guide. Greensboro, NC: Center for Creative Leadership.
19.
London, M., & Smither, J. W. (1995). Can multi-source feedback change perceptions of goal accomplishment, self-evaluations, and performance-related outcomes? Theory-based applications and directions for research. Personnel Psychology, 48, 803-839.
20.
Maurer, T. J., Raju, N. S., & Collins, W. C. (1998). Peer and subordinate performance appraisal measurement equivalence. Journal of Applied Psychology, 83, 693-702.
21.
McCauley, C. D., & Lombardo, M. M. (1990). Benchmarks: An instrument for diagnosing managerial strengths and weaknesses. In K. E. Clark & M. B. Clark (Eds.), Measures of leadership. West Orange, NJ: Leadership Library of America.
22.
McCauley, C. D., Lombardo, M. M., & Usher, C. J. (1989). Diagnosing management development needs: An instrument based on how managers develop. Journal of Management, 15, 389-403.
23.
McDaniel, M. A., Whetzel, D. L., Schmidt, F. L. & Maurer, S. D. (1994). The validity of employment interviews: A comprehensive review and meta-analysis. Journal of Applied Psychology, 79, 599-616.
24.
Mislevy, R. J., & Bock, R. D. (1990). Bilog 3 user manual. Mooresville, IN: Scientific Software.
25.
Mount, M. K., Judge, T. A., Scullen, S. E., Sytsma, M. R., & Hezlett, S. A. (1998). Trait, rater and level effects in 360-degree performance ratings. Personnel Psychology, 51, 557-576.
26.
Raju, N. S. (1999a). DFIT4P: A computer program for analyzing differential item and test functioning. Chicago: Illinois Institute of Technology.
27.
Raju, N. S. (Chair). (1999b, April). IRT-based evaluation of 360° feedback assessments: The BENCHMARKS story. Symposium presented at the annual conference of the Society for Industrial-Organizational Psychology, Atlanta, GA.
28.
Raju, N. S., Leslie, J. B., McDonald-Mann, D., & Craig, S. B. (2001). Measurement equivalence of a 360-degree feedback assessment. Unpublished manuscript.
29.
Raju, N. S., van der Linden, W., & Fleer, P. (1995). An IRT-based internal measure of test bias with applications for differential item functioning. Applied Psychological Measurement, 19, 353-368.
30.
Reckase, M. D. (1979). Unifactor latent trait models applied to multi-factor tests: Results and implications. Journal of Educational Statistics, 4, 207-230.
31.
Samejima, F. (1969). Estimation of latent ability using a response pattern of graded scores (Psychometrika Monograph No. 17).
32.
Schriesheim, C. A., House, R. J., & Kerr, S. (1976). Leader initiating structure: A reconciliation of discrepant research results and some empirical tests. Organizational Behavior and Human Performance, 15, 297-321.
33.
Scullen, S. E., Mount, M. K., & Goff, M. (2000). Understanding the latent structure of job performance ratings. Journal of Applied Psychology, 85, 956-970.
34.
Stocking, M. L., & Lord, F. M. (1983). Developing a common metric in item response theory. Applied Psychological Measurement, 7, 201-210.
35.
Supovitz, J. A., MacGowan, A., Slattery, J. (1997). Assessing agreement: An examination of the interrater reliability of portfolio assessment in Rochester, New York. Educational Assessment, 4, 237-259.
36.
Thissen, D. (1995). MULTILOG 6.3: A computer program for multiple, categorical item analysis and test scoring using item response theory. Chicago: Scientific Software.
37.
Thornton, G. C. (1992). Assessment centers in human resource management. Reading, MA: Addison-Wesley.
38.
Tornow, W. (1993). Perceptions or reality: Is multi-perspective measurement a means or an end? Human Resource Management, 32, 221-230.
39.
Tornow, W. W., & London, M. (1998). Maximizing the value of 360-degree feedback. San Francisco: Jossey-Bass.
40.
Zedeck, S. (1995). [Review of benchmarks]. In J. Conoley & J. Impara (Eds.), The twelfth mental measurements yearbook (Vol. 1, pp. 128-129). Lincoln, NE: Buros Institute of Mental Measurements.
