A Revised Index of Interrater Agreement for Multi-Item Ratings of a Single Target

Abstract

The commonly used form of r _{wg. (J)} can display irregular behavior, so four variants of this index were examined. An alternative index, r*_{wg. J}, is recommended. This index is an inverse linear function of the ratio of the average obtained variance to the variance of uniformly distributed random error. r*_wg.Jis superficially similar to Cronbach’s α, but careful examination confirms that r*_wg.Jis an index of agreement, not reliability. Based on an examination of the small-sample behavior of r _wgand r*_wg.J, sample sizes of 10 or more raters are recommended.

Get full access to this article

View all access options for this article.

References

Bailey, D. E. (1971). Probability and statistics: Models for research. New York: Wiley.

Cronbach, L. J. (1951). Coefficient α and the internal structure of tests. Psychometrika, 16, 297–334.

Finn, R. H. (1970). Anote on estimating the reliability of categorical data. Educational and Psychological Measurement, 30, 71–76.

James, L. R. , Demaree, R. G. , & Wolfe, G. (1984). Estimating within-group interrater reliability with and without response bias. Journal of Applied Psychology, 69, 85–98.

James, L. R. , Demaree, R. G. , & Wolfe, G. (1993). rwg: An assessment of within-group interrater agreement. Journal of Applied Psychology, 78, 306–309.

Kozlowski, S. W. J. & Hattrup, K. (1992). A disagreement about within-group agreement: Disentangling issues of consistency versus consensus. Journal of Applied Psychology, 77, 161–167.

Lawlis, G. F. & Lu, E. (1972). Judgment of counseling process: Reliability, agreement, and error. Psychological Bulletin, 78, 17–20.

Lindell, M. K. & Brandt, C. J. (1997). Measuring interrater agreement for ratings of a single target. Applied Psychological Measurement, 21, 271–278.

Phillips, L. D. (1974). Bayesian statistics for social scientists. New York: Crowell.

10.

Schmidt, F. L. & Hunter, J. E. (1989). Interrater reliability coefficients cannot be computed when only one stimulus is rated. Journal of Applied Psychology, 74, 368–370.

11.

Tinsley, H. E. A. & Weiss, D. J. (1975). Interrater reliability and agreement of subjective judgments. Journal of Counseling Psychology, 22, 358–376.