An index for assessing interrater agreement with respect to a single target using a multi-item rating scale is developed. The variance of rater mean scale scores is used as the numerator of the agreement index. Four variants of a disattenuated agreement index that vary in the random response term used as the denominator are examined. A variant of rwg(J)using perfectly collinear, uniformly distributed responses as the baseline is recommended as a useful index of disattenuated agreement. A test of the equality of variances is recommended for deleting items with a level of interrater agreement significantly different from zero but lower than that of the remaining items.
Get full access to this article
View all access options for this article.
References
1.
Cronbach, L. J. (1951). Coefficient α and the internal structure of tests. Psychometrika, 16,297–334.
2.
Glass, G. V. , & Stanley, J. C. (1970). Statistical methods in education and psychology.Englewood Cliffs NJ: Prentice-Hall.
3.
James, L. R. , Demaree, R. G., & Wolfe, G. (1984). Estimating within-group interrater reliability with and without response bias. Journal of Applied Psychology, 69,85–98.
4.
Lindell, M. K. , & Brandt, C. J. (1997). Measuring interrater agreement for ratings of a single target. Applied Psychological Measurement, 21,271–278.
5.
Lindell, M. K. , & Brandt, C. J. (1999). Assessing interrater agreement on the job relevance of a test: A comparison of the CVI, T, rWG(j), and r* WGindexes. Journal of Applied Psychology, 84,640–647.
6.
Lindell, M. K. , Brandt, C. J., & Whitney, D. J. (1999). A revised index of interrater agreement for multiitemratings of a single target. Applied Psychological-Measurement, 23,127–135.
7.
Tinsley, H. E. A. , & Weiss, D. J. (1975). Interrater reliability and agreement of subjective judgments. Journal of Counseling Psychology, 22,358–376.
