The accuracy of the large sample standard error of weighted kappa appropriate to the non-null case was studied by computer simulation. Results indi cate that only moderate sample sizes are required to test the hypothesis that two independently de rived estimates of weighted kappa are equal. How ever, in most instances the minimal sample sizes re quired for setting confidence limits around a single value of weighted kappa are inordinately large. An alternative, but as yet untested procedure for set ting confidence limits, is suggested as being poten tially more accurate.
Get full access to this article
View all access options for this article.
References
1.
Cicchetti, D.V.A new measure of agreement between rank ordered variables. Proceedings of the American Psychological Association, 1972 , 7, 17-18.
2.
Cicchetti, D.V., & Fleiss, J.L.Comparison of the null distributions of weighted kappa and the C ordinal statistic . Applied Psychological Measurement, 1977, 1, 195-201.
3.
Cohen, J.Weighted kappa: nominal scale agreement with provision for scaled disagreement or partial credit. Psychological Bulletin, 1968, 70, 213-220.
4.
Fleiss, J.L., & Cohen, J.The equivalence of weighted kappa and the intraclass correlation coefficient as measures of reliability. Educational and Psychological Measurement, 1973, 33, 613-619.
5.
Fleiss, J.L., Cohen, J., & Everitt, B.S.Large sample standard errors of kappa and weighted kappa. Psychological Bulletin, 1969, 73, 323-327.
