Sage Journals: Discover world-class research

Abstract

A recent article by Lindell and Brandt raised two concerns regarding the use of James, Demaree, and Wolf’s interrater agreement indices r_WG and r_WG(J). First, they noted that the multi-item _rWG(J) equation is mathematically equivalent to inserting r_WG into the Spearman-Brown prophecy formula and questioned whether applying a formula developed for a reliability index was also appropriate for an agreement index. Second, they questioned the appropriateness of James et al.’s suggestion of replacing obtained negative values of r_WG with zeros. This article addresses these concerns by demonstrating that r_WG(J) can be derived independently from the Spearman-Brown prophecy formula and that negative values of r_WG can be avoided by reparameterizing the structural equation underlying the data when there is systematic disagreement between subgroups of raters.

Keywords

interrater agreement interrater disagreement Spearman-Brown rWG

Get full access to this article

View all access options for this article.

References

Cohen, A. , Doveh, E. , & Eick, U. (2001). Statistical properties of the rWG(J) index of agreement. Psychological Methods, 6, 297-310.

Gulliksen, H. (1950). Theory of mental tests. New York: John Wiley.

Hays, W. L. (1988). Statistics (4th ed.). New York: Holt, Rinehart & Winston.

James, L. R. , Demaree, R. G. , & Wolf, G. (1984). Estimating within-group interrater reliability with and without response bias. Journal of Applied Psychology, 69, 85-98.

James, L. R. , Demaree, R. G. , & Wolf, G. (1993). rWG: An assessment of within-group interrater agreement. Journal of Applied Psychology, 78, 306-309.

Kozlowski, S. W. J. , & Hattrup, K. (1992). A disagreement about within-group agreement: Disentangling issues of consistency versus consensus. Journal of Applied Psychology, 77, 161-167.

Lindell, M. K. (2001). Assessing and testing interrater agreement in multi-item rating scales. Applied Psychological Measurement, 25, 89-99.

Lindell, M. K. , & Brandt, C. J. (1997). Measuring interrater agreement for ratings of a single target. Applied Psychological Measurement, 21, 271-278.

Lindell, M. K. , & Brandt, C. J. (1999). Assessing interrater agreement on the job relevance of a test: A comparison of the CVI, T, rWG(J), and r*WG indexes. Journal of Applied Psychology, 84, 640-647.

10.

Lindell, M. K. , Brandt, C. J. , & Whitney, D. J. (1999). A revised index of agreement for multi-item ratings of a single target. Applied Psychological Measurement, 23, 127-135.

Recent Issues Regarding rWG,rWG,rWG(J),and rWG(J)

Abstract

Keywords

Get full access to this article

References