Estimation of Quantiles for the Sampling Distribution of the rWG within Group Agreement Index

Abstract

The James, Demaree, and Wolf within-group agreement index (rwG) may be used to assess interrater agreement when two or more individuals ("judges") provide ratings on one or more items ("targets"). Although the James et al. index is increasingly being used to assess whether the aggregation of data is justified for higher levels of analysis, the sampling distribution of rWG has not been delineated, and therefore it has not been possible to conduct tests of the statistical significance of obtained results. As a consequence of this current deficiency in knowledge about rWG, a Monte Carlo simulation program (AGREESIM) was developed to estimate selected quantiles for the sampling distribution of rWG under any user-specified null distribution of judges' scores. The output is in the form of quantiles and their 95% confidence intervals (showing the precision of the quantile estimates).

Get full access to this article

View all access options for this article.

References

George, J. M. (1990). Personality, affect, and behavior in groups. Journal of Applied Psychology, 75, 107-116.

George, J. M. , & James, L. R. (1993). Personality, affect, and behavior in groups revisited: Comment on aggregation, levels of analysis, and a recent application of within and between analysis. Journal of Applied Psychology, 78, 798-804.

James, L. R. , Demaree, R. G. , & Wolf, G. (1984). Estimating within-group interrater reliability with and without response bias. Journal of Applied Psychology, 69, 85-98.

Kozlowski, S.W.J. , & Hattrup, K. (1992). A disagreement about within-group agreement: Disentangling issues of consistency versus consensus. Journal of Applied Psychology, 77, 161-167.

Yammarino, F. J. , & Markham, S. E. (1992). On the application of within and between analysis: Are absence and affect really group-based phenomena? Journal of Applied Psychology, 77, 168-176.