Agreement Lambda for Weighted Disagreement With Ordinal Scales: Correction for Category Prevalence

Abstract

Weighted inter-rater agreement allows for differentiation between levels of disagreement among rating categories and is especially useful when there is an ordinal relationship between categories. Many existing weighted inter-rater agreement coefficients are either extensions of weighted Kappa or are formulated as Cohen’s Kappa-like coefficients. These measures suffer from the same issues as Cohen’s Kappa, including sensitivity to the marginal distributions of raters and the effects of category prevalence. They primarily account for the possibility of chance agreement or disagreement. This article introduces a new coefficient, weighted Lambda, which allows for the inclusion of varying weights assigned to disagreements. Unlike traditional methods, this coefficient does not assume random assignment and does not adjust for chance agreement or disagreement. Instead, it modifies the observed percentage of agreement while taking into account the anticipated impact of prevalence-agreement effects. The study also outlines techniques for estimating sampling standard errors, conducting hypothesis tests, and constructing confidence intervals for weighted Lambda. Illustrative numerical examples and Monte Carlo simulations are presented to investigate and compare the performance of the new weighted Lambda with commonly used weighted inter-rater agreement coefficients across various true agreement levels and agreement matrices. Results demonstrate several advantages of the new coefficient in measuring weighted inter-rater agreement.

Keywords

weighted Lambda weighted Kappa weighted disagreement prevalence-agreement effect

Get full access to this article

View all access options for this article.

References

Almehrizi

R. S.

(2025). Coefficient of agreement between two raters corrected for category prevalence: Alternative to kappa. Psychological Methods. Advance online publication. https://doi.org/10.1037/met0000732

Brennan

R. L.

Prediger

D. J.

(1981). Coefficient kappa: Some uses, misuses, and alternatives. Educational and Psychological Measurement, 41, 687–699. https://doi.org/10.1177/001316448104100307

Cicchetti

D. V.

(1976). Assessing interrater reliability for rating scales: Resolving some basic issues. British Journal of Psychiatry, 129, 452–456.

Cohen

(1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20, 37–46. https://doi.org/10.1177/001316446002000104

Cohen

(1968). Weighted kappa: Nominal scale agreement with provision for scaled disagreement or partial credit. Psychological Bulletin, 70, 213–220.

Erdoğan

Suc

(2023). Status of weighted agreement statistics between two raters in ordinal data affected by sample size and number of categories. Euroasia Journal of Mathematics, Engineering, Natural & Medical Sciences, 28, 219–231.

Fleiss

Cohen

(1973). The equivalence of weighted kappa and the intraclass correlation coefficient as measures of reliability. Educational and Psychological Measurement, 33, 613–169.

Fleiss

J. L.

Cohen

Everitt

B. S.

(1969). Large sample standard errors of kappa and weighted kappa. Psychological, Bulletin, 72(5), 323–327. https://doi.org/10.1037/h0028106

Graham

Jackson

(1993). The analysis of ordinal agreement data: Beyond weighted kappa. Journal of Clinical Epidemiology, 46, 1055–1062.

10.

Gwet

K. L.

(2021). Handbook of inter-rater reliability. StatAxis.

11.

Gwet

K. L.

(2008). Computing inter-rater reliability and its variance in the presence of high agreement. British Journal of Mathematical and Statistical Psychology, 61(1), 29–48.

12.

Gwet

K. L.

(2014). Handbook of inter-rater reliability: The definitive guide to measuring the extent of agreement among raters (4th ed.). Advance Analytics.

13.

Heo

(2008). Utility of weights for weighted kappa as a measure of interrater agreement on ordinal scale. Journal of Modern Applied Statistical Methods, 7, 205–222. https://doi.org/10.22237/jmasm/1209615360

14.

Hubert

L. J.

(1978). A general formula for the variance of Cohen’s weighted kappa. Psychological Bulletin, 85, 183–184.

15.

Krippendorff

(1978). Reliability of binary attribute data. Biometrics, 34(1), 142–144.

16.

Krippendorff

(2004). Measuring the reliability of qualitative text analysis data. Quality & Quantity, 38(6), 787–800.

17.

Landis

J. R.

Koch

G. G.

(1977). The measurement of observer agreement for categorical data. Biometrics, 33, 159–174. https://doi.org/10.2307/2529310

18.

Maclure

Willett

W. C.

(1987). Misinterpretation and misuse of the kappa statistic. American Journal of Epidemiology, 126(2), 161–169.

19.

Minozzi

Cinquini

Gianola

Gonzalez-Lorenzo

Banzi

(2022). Kappa and AC1/2 statistics: Beyond the paradox. Journal of Clinical Epidemiology, 142, 328–329. https://doi.org/10.1016/j.jclinepi.2021.09.004

20.

Muthén

(1984). A general structural equation model with dichotomous, ordered categorical, and continuous latent variable indicators. Psychometrika, 49(1), 115–132.

21.

Nelson

J. C.

Pepe

M. S.

(2000). Statistical description of interrater variability in ordinal ratings. Statistical Methods in Medical Research, 9, 475–496.

22.

Rao

C. R.

(1948). Large sample tests of statistical hypotheses concerning several parameters with applications to problems of estimation. Proceedings of the Cambridge Philosophical Society, 44, 50–57.

23.

Schuster

(2004). A Note on the interpretation of weighted Kappa and its relations to other rater agreement statistics for metric scales. Educational and Psychological Measurement, 64(2), 243–253. https://doi.org/10.1177/0013164403260197

24.

Scott

(1955). Reliability of content analysis: The case of nominal scale coding. Public Opinion Quarterly, 19(3), 321–325.

25.

Shrout

P. E.

(1998). Measurement reliability and agreement in psychiatry. Statistics Methods in Medical Research, 7, 301–317.

26.

Sim

Wright

(2005). The kappa statistic in reliability studies: Use, interpretation, and sample size requirements. Physical Therapy, 85, 257–268.

27.

Tran

Dolgun

Demirhan

(2020). Weighted inter-rater agreement measures for ordinal outcomes. Communications in Statistics: Simulation and Computation, 49, 989–1003. https://doi.org/10.1080/03610918.2018.1490428

28.

Vach

Gerke

(2023). Gwet’s AC1 is not a substitute for Cohen’s kappa: A comparison of basic properties. Methodsx, 10, 1–8. https://doi.org/10.1016/j.mex.2023.102212

29.

Warrens

M. J.

(2012). Some paradoxical results for the quadratically weighted kappa. Psychometrika, 77(2), 315–323.

30.

Warrens

M. J.

(2017). Symmetric kappa as a function of unweighted kappas. Communications in Statistics: Simulation and Computation, 46, 1–6.

31.

Wongpakaran

Wedding

Gwet

K. L.

(2013). A comparison of Cohen’s kappa and Gwet’s AC1 when calculating inter-rater reliability coefficients: A study conducted with personality disorder samples. BMC Medical Research Methodology, 13, Article 61. https://doi.org/10.1186/1471-2288-13-61

32.

Yilmaz

A. E.

Saracbasi

(2017). Assessing agreement between raters from the point of coefficients and log-linear models. Journal of Data Science, 15(1), 1–24.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.18 MB