The kappa agreement coefficient of Cohen from 1960 and Brennan and Prediger from 1981 are defined and compared. A FORTRAN program is described that computes Cohen's kappa and Brennan and Prediger's kappa and their associated probability values based on Monte Carlo resampling and the binomial distribution, respectively.
Get full access to this article
View all access options for this article.
References
1.
BartkoJ. J. (1991) Measurement and reliability—statistical thinking considerations. Schizophrenia Bulletin, 17, 483–489.
2.
BennettE. M.AlpertR.GoldsteinA. C. (1954) Communications through limited-response questioning. Public Opinion Quarterly, 18, 303–308.
3.
BerryK. J.MielkeP. W.Jr. (1988) A generalization of Cohen's kappa agreement measure to interval measurement and multiple raters. Educational and Psychological Measurement, 48, 921–933.
4.
BerryK. J.MielkeP. W.Jr.HelmericksS. G. (1994) An algorithm to generate discrete probability distributions: hypergeometric, negative binomial inverse hypergeometric, and Poisson. Behavior Research Methods. Instruments, & Computers, 26, 366–367.
5.
BrennanR. L.PredigerD. J. (1981) Coefficient kappa: some uses, misuses, and alternatives. Educational and Psychological Measurement, 41, 687–699.
6.
CohenJ. (1960) A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20, 37–46.
7.
CollisG. M. (1985) Kappa, measures of marginal symmetry and intraclass correlations. Educational and Psychological Measurement, 45, 55–62.
8.
FleissI. L.CohenJ.EverittB. S. (1969) Large samples standard errors of kappa and weighted kappa. Psychological Bulletin, 72, 323–327.
9.
HanllyJ. A. (1987) Standard error of the kappa statistic. Psychological Bulletin, 102, 315–321.
10.
HolleyJ. W.GuilfordJ. P. (1964) A note on the G index of agreement. Educational and Psychological Measurement, 4, 749–753.
11.
JansonS.VegeliusJ. (1979) On generalizations of the G index and the phi coefficient to nominal scales. Multivariate Behavioral Research, 14, 255–269.
MaxwellA. E. (1977) Coefficients of agreement between observers and their interpretation. British Journal of Psychiatry, 130, 79–83.
14.
MeyerG. J. (1997) Assessing reliability: critical corrections for a critical examination of the Rorschach Comprehensive System. Psychological Assessment, 9, 480–489.
15.
ShroutP. E.SpitzerR. L.FleissJ. L. (1987) Quantification of agreement in psychiatric diagnosis revisited. Archives of General Psychiatry, 44, 172–177.
16.
UmeshU. N.PetersonR. A.SauberM. H. (1989) Interjudge agreement and the maximum value of kappa. Educational and Psychological Measurement, 49, 835–850.
17.
ZwickR. (1988) Another look at interrater agreement. Psychological Bulletin, 103, 374–378.
