We examined the extent to which performance ratings obtained using a distributional rating format differ from ratings obtained using a more typical global rating scale format. We sought to determine whether a distributional format actually elicits different information than a global format and the extent to which measures of within performance variability derived from distributional ratings are related to levels of interrater agreement on global rating scales. Confirmatory factor analyses of three models of measurement equivalence indicated that a model of tau-equivalence in which the two rating formats provided similar measurements but with differing levels of measurement error provided the best fit to the data. Further, we found a large, significant relationship between measures of within dimension performance variabili5 and interrater agreement. Our results suggest that a distributional rating format is a viable alternative rating format that may provide a richer source of information than do more typical global rating scales.
Get full access to this article
View all access options for this article.
References
1.
Blanz. F. & Ghiselli, E.E. (1972). The mixed standard scale: A new rating system. Personnel Psychology, 25: 185-199.
2.
Browne, M. W. (1977). Generalized least-squares estimators in the analysis of covariance structures, (pp. 205-266) in D.J. Aigner & A.S. Goldberger (Eds.), Latent variables in socio-economic models. Amsterdam: North-Holland.
3.
Hambleton, R.K., Swaminathan. H. & Rodgers. H.J. (1991). Fundamentals of Item Response Theory. Newbury Park, CA.: Sage Publications Inc.
4.
Hattrup, K., Schmitt, N. & Landis, R.S. (1992). Equivalence of constructs measured by job-specific and commercially available aptitude tests. Journal of Applied Psychology, 77: 298-308.
5.
Holland, J.H., Holyoak, K.J., Nisbett, R.E. & Thagard, P.R. (1986). Induction: Processes of inference, learning, and discovery. Cambridge, MA: MIT Press.
6.
Jako, R.A. & Murphy, K.R. (1990). Distributional ratings, judgment decomposition, and their impact on interrater agreement and rating accuracy. Journal of Applied Psychology, 75: 500-505.
7.
James, L.R., Demaree, R.G. & Wolf, G. (1984). Estimating within-group interrater reliability with and without response bias. Journal of Applied Psychology, 69: 85-98.
8.
Jöreskog, K. G. (1969). A general approach to confirmatory maximum likelihood factor analysis. Psychometrika, 34: 183-202.
9.
Jöreskog, K. G. & Sörbom, D. (1989). LISREL VII: A guide to the program and applications (2nd edition). Chicago: SPSS, Inc.
10.
Jöreskog, K. G. & Sörbom, D. (1993). LISREL-8: Structural Equation Modeling with the SIMPLIS Command Language. Chicago, IL: Scientific Software International Inc.
11.
Kane, J.S. (1983). Performance distribution assessment: A new breed of appraisal methodology, (pp. 325-341) in H.J. Bemardin & R.W. Beatty (Eds.), Performance appraisal: Assessing human behavior at work. Boston: Kent.
12.
Kane, J.S. (1986). Performance distribution assessment, (pp. 237-273) in R.A. Berk (Ed.), Performance assessment. Baltimore, MD: Johns Hopkins University Press.
13.
Kane, J.S. & Lawler, E.E. (1979). Performance appraisal effectiveness: Its assessment and determinants, (pp. 425-478) in B. Staw (Ed.), Research in organizational behavior, (Vol. 1). Greenwich, CT: JAI Press.
Nunnally, J. C. & Bernstein, I. H. (1994). Psychometric theory. New York: McGraw-Hill.
17.
Smith, P.C. & Kendall, L.M. (1963). Retranslation of expectations: An approach to the construction of unambiguous anchors for rating scales. Journal of Applied Psychology, 447: 149-155.
18.
Steiner, D.D., Rain, J.S. & Smalley, M.M. (1993). Distributional ratings of performance: Further examination of a new rating format. Journal of Applied Psychology, 78: 438-442.
19.
Turban, D. B., Sanders, P. A., Francis, D. J. & Osburn, H. G. (1989). Construct equivalence as an approach to replacing validated cognitive ability tests. Journal of Applied Psychology, 74: 62-71.
