Comparison of the Null Distributions of Weighted Kappa and the C Ordinal Statistic

Abstract

It frequently occurs in psychological research that an investigator is interested in assessing the ex tent of interrater agreement when the data are measured on an ordinal scale. This monte carlo study demonstrates that the appropriate statistic to apply is weighted kappa with its revised standard error. The study also demonstrates that the mini mal number of cases required for the valid applica tion of weighted kappa varies between 20 and 100, depending upon the size of the ordinal scale. This contrasts with a previously cited large sample esti mate of 200. Given the difficulty of obtaining sam ple sizes this large, the latter finding should be of some comfort to investigators who use weighted kappa to measure interrater consensus.

Get full access to this article

View all access options for this article.

References

Cicchetti, D.V. A new measure of agreement between rank ordered variables. Proceedings of the American Psychological Association. 1972, 7. 17-18. (a)

Cicchetti, D.V. Assessing observer agreement with dichotomous-ordinalized data. Biometrics. 1972, 28. 1165 (Abstract). (b)

Cohen, J. Weighted Kappa: nominal scale agreement with provision for scaled disagreement or partial credit. Psychological Bulletin, 1968, 70. 213-220.

Fleiss, J.L. , & Cohen, J. The equivalence of weighted kappa and the intraclass correlation coefficient as measures of reliability. Educational and Psychological Measurement, 1973, 33, 613-619.

Fleiss, J.L. , Cohen, J. , & Everitt, B.S. Large sample standard errors of kappa and weighted kappa. Psychological Bulletin. 1969, 72. 323-327.

Krippendorff, K. Bivariate agreement coefficients for reliability of data. In E. G. Borgatta (Ed.), Sociological Methodology . San Francisco: Jossey-Bass . 1970.