Despite its well-known weaknesses, researchers continuously choose the kappa coefficient (Cohen, 1960, Educational and Psychological Measurement 20: 37–46; Fleiss, 1971, Psychological Bulletin 76: 378–382) to quantify agreement among raters. Part of kappa's persistent popularity seems to arise from a lack of available alternative agreement coefficients in statistical software packages such as Stata. In this article, I review Gwet’s (2014, Handbook of Inter-Rater Reliability) recently developed framework of interrater agreement coefficients. This framework extends several agreement coefficients to handle any number of raters, any number of rating categories, any level of measurement, and missing values. I introduce the kappaetc command, which implements this framework in Stata.
AltmanD. G.1991. Practical Statistics for Medical Research.London: Chapman & Hall.
2.
BennettE. M., AlpertR., and GoldsteinA. C.1954. Communications through limited-response questioning. Public Opinion Quarterly18: 303–308.
3.
BlandJ. M., and AltmanD. G.1986. Statistical methods for assessing agreement between two methods of clinical measurement. Lancet327: 307–310.
4.
BrennanR. L., and PredigerD. J.1981. Coefficient kappa: Some uses, misuses, and alternatives. Educational and Psychological Measurement41: 687–699.
5.
ByrtT., BishopJ., and CarlinJ. B.1993. Bias, prevalence and kappa. Journal of Clinical Epidemiology46: 423–429.
6.
CohenJ.1960. A coefficient of agreement for nominal scales. Educational and Psychological Measurement20: 37–46.
7.
CohenJ.1968. Weighted kappa: Nominal scale agreement provision for scaled disagreement or partial credit. Psychological Bulletin70: 213–220.
8.
CongerA. J.1980. Integration and generalization of kappas for multiple raters. Psychological Bulletin88: 322–328.
9.
CoxN. J.2016. entropyetc: Stata module for entropy and related measures for categories. Statistical Software Components S458272, Department of Economics, Boston College. https://ideas.repec.org/c/boc/bocode/s458272.html.
10.
FeinsteinA. R., and CicchettiD. V.1990. High agreement but low kappa: I. The problems of two paradoxes. Journal of Clinical Epidemiology43: 543–549.
11.
FengG. C.2013. Factors affecting intercoder reliability: A Monte Carlo experiment. Quality and Quantity47: 2959–2982.
12.
FleissJ. L.1971. Measuring nominal scale agreement among many raters. Psychological Bulletin76: 378–382.
13.
FleissJ. L., CohenJ., and EverittB. S.1969. Large sample standard errors for kappa and weighted kappa. Psychological Bulletin72: 323–327.
14.
FleissJ. L., LevinB., and PaikM. C.2003. Statistical Methods for Rates and Proportions.3rd ed.Hoboken, NJ: Wiley.
15.
GwetK. L.2008a. Computing inter-rater reliability and its variance in the presence of high agreement. British Journal of Mathematical and Statistical Psycholgy61: 29–48.
16.
GwetK. L.2008b. Variance estimation of nominal-scale inter-rater reliability with random selection of raters. Psychometrika73: 407–430.
17.
GwetK. L.2014. Handbook of Inter-Rater Reliability: The Definitive Guide to Measuring the Extent of Agreement Among Raters.4th ed.Gaithersburg, MD: Advanced Analytics.
GwetK. L.2016. Testing the difference of correlated agreement coefficients for statistical significance. Educational and Psychological Measurement76: 609–637.
20.
HarrisonD.2004. kaputil: Stata module to generate confidence intervals and sample size calculations for the kappa-statistic. Statistical Software Components S446501, Department of Economics, Boston College. https://ideas.repec.org/c/boc/bocode/s446501.html.
21.
HayesA. F., and KrippendorffK.2007. Answering the call for a standard reliability measure for coding data. Communication Methods and Measures1: 77–89.
22.
KleinD.2014. kalpha: Stata module to compute Krippendorff's alpha-reliability. Statistical Software Components S457862, Department of Economics, Boston College. https://ideas.repec.org/c/boc/bocode/s457862.html.
23.
KrippendorffK.1970. Estimating the reliability, systematic error and random error of interval data. Educational and Psychological Measurement30: 61–70.
KrippendorffK.2013. Content Analysis: An Introduction to Its Methodology.3rd ed.Thousand Oaks, CA: Sage.
26.
LandisJ. R., and KochG. G.1977. The measurement of observer agreement for categorical data. Biometrics33: 159–174.
27.
LazaroJ., ZamoraJ., AbrairaV., and ZlotnikA.2013. kappa2: Stata module to produce generalizations of weighted kappa for incomplete designs. Statistical Software Components S457739, Department of Economics, Boston College. https://ideas.repec.org/c/boc/bocode/s457739.html.
28.
MitnikP.2016. kanom: Stata module to estimate Krippendorff's alpha for nominal variables. Statistical Software Components S458277, Department of Economics, Boston College. https://ideas.repec.org/c/boc/bocode/s458277.html.
29.
MitnikP., and CumberworthE.2016. Measuring social class with changing occupational classifications: Reliability, competing measurement strategies, and the 1970–1980 U.S. classification divide. Working Paper, Stanford Center on Poverty and Inequality. https://web.stanford.edu/∼pmitnik/Mitnik_Cumberworth_2016.pdf.
30.
ReichenheimM. E.2004. Confidence intervals for the kappa statistic. Stata Journal4: 421–428.
31.
ScottW. A.1955. Reliability of content analysis: The case of nominal scale coding. Public Opinion Quarterly19: 321–325.
32.
StaudtA., and KrewelM.2013. krippalpha: Stata module to compute Krippendorff's alpha intercoder reliability coefficient. Statistical Software Components S457750, Department of Economics, Boston College. https://ideas.repec.org/c/boc/bocode/s457750.html.
33.
WarrensM. J.2012. Some paradoxical results for the quadratically weighted kappa. Psychometrika77: 315–323.
34.
WarrensM. J.2014. Power weighted versions of Bennett, Alpert, and Goldstein's S. Journal of Mathematics2014: 231909.
35.
WarrensM. J., and PratiwiB. C.2016. Kappa coefficients for circular classifications. Journal of Classification33: 507–522.
36.
WongpakaranN., WongpakaranT., WeddingD., and GwetK. L.2013. A comparison of Cohen's kappa and Gwet's AC1 when calculating inter-rater reliability coefficients: A study conducted with personality disorder samples. BMC Medical Research Methodology13: 61.
37.
ZapfA., CastellS., MorawietzL., and KarchA.2016. Measuring inter-rater reliability for nominal data—Which coefficients and confidence intervals are appropriate?BMC Medical Research Methodology16: 93.
