The main purposes of the present article are (a) to exemplify a link between Fleiss's multiple-rater kappa or other analogues and the generalizability (G) coefficient for a single facet design, (b) to explore the possible utility and interpretation of G theory in the study of interrater agreement when the data are measured on a nominal scale, and (c) to explain why the G coefficient is preferred in place of the kappa coefficient and its derived forms.
Get full access to this article
View all access options for this article.
References
1.
Brennan, R. L. (1992). Elements of generalizability theory (Rev. ed.). Iowa City, IA: ACT.
2.
Cicchetti, D V. , & Sparrow, S. A. (1981). Developing criteria for establishing interrater reliability of specific items: Applications to assessment of adaptive behavior. American Journal of Mental Deficiency, 86, 127-137.
3.
Collis, G. M. (1985). Kappa, measures of marginal symmetry and interclass correlations. Educational and Psychological Measurement, 45, 55-62.
4.
Fleiss, J. L. (1971). Measuring nominal scale agreement among many raters. Psychological Bulletin, 76, 378-382.
5.
Fleiss, J. L. (1981). Statistical methods for rates and proportions (2nd ed.). New York: John Wiley.
6.
Gini, C. (1939). Variabilita e concentrazione. Vol 1: Memorie di metodologia statistica [Variability and concentration. Vol. 1: Notes of statistical methodology]. Milano, Italy: Giuffre.
7.
Landis, J. R. , & Koch, G. (1977). A one-way components of variance model for categorical data. Biometrics, 33, 671-679.
8.
Li, M. N. (1996). A SAS macro program for interrater agreement studies of qualitative data: A generalizability approach. Manuscript submitted for publication.
9.
Light, R. J. , & Margolin, B. H. (1971). An analysis of variance for category data. Journal of the American Statistical Association, 66, 534-544.
10.
Rae, G. (1988). The equivalence of multiple rater kappa statistics and intraclass correlation coefficients. Educational and Psychological Measurement, 48, 367-374.
11.
Shavelson, R. J. , & Webb, N. M. (1991). Generalizability theory: A primer. Newbury Park, CA: Sage.
12.
Suen, H. K. , & Ary, D. (1989). Analyzing quantitative behavioral observation data. Hillsdale, NJ: Lawrence Erlbaum.
