A Tool for Agreement and Alignment Analysis in Binary Rating Tasks: The R Package scindex

Abstract

scindex is an R package for analysing inter-rater reliability in binary classification tasks. The package computes Cohen’s κ and Fleiss’ κ and, when ground-truth labels are available, estimates signal detection theory parameters, including sensitivity, specificity, and decision thresholds. It also implements the Strategic Convergence Index (SCI), a measure of convergence in raters’ response criteria.

Keywords

inter-rater reliability signal detection theory binary classification decision thresholds agreement analysis R

Get full access to this article

View all access options for this article.

References

Byrt

Bishop

Carlin

J. B.

(1993). Bias, prevalence and kappa. Journal of Clinical Epidemiology, 46(5), 423–429. https://doi.org/10.1016/0895-4356(93)90018-V

Cohen

(1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1), 37–46. https://doi.org/10.1177/001316446002000104

DeCarlo

L. T.

(1998). Signal detection theory and generalized linear models. Psychological Methods, 3(2), 186–205. https://doi.org/10.1037/1082-989X.3.2.186

Feinstein

A. R.

Cicchetti

D. V.

(1990). High agreement but low kappa: I. The problems of two paradoxes. Journal of Clinical Epidemiology, 43(6), 543–549. https://doi.org/10.1016/0895-4356(90)90158-L

Fleiss

J. L.

(1971). Measuring nominal scale agreement among many raters. Psychological Bulletin, 76(5), 378–382. https://doi.org/10.1037/h0031619

Gianeselli

(2026a). scindex: Strategic convergence index for inter-rater reliability (Version 0.1.0) [R package]. Comprehensive R Archive Network. https://doi.org/10.32614/CRAN.package.scindex

Gianeselli

(2026b). From agreement to epistemic alignment: A signal detection–theoretic model of inter-rater reliability. Educational and Psychological Measurement. Advance online publication. https://doi.org/10.1177/00131644261417643

Gianeselli

(2026c). Agreement and alignment in binary rating tasks: Strategic convergence as an equilibrium outcome. Educational and Psychological Measurement. Advance online publication. https://doi.org/10.1177/00131644261455307

Green

D. M.

Swets

J. A.

(1966). Signal detection theory and psychophysics. Wiley.

10.

R Core Team . (2026). R: A language and environment for statistical computing. R Foundation for Statistical Computing. https://www.r-project.org/

11.

Warrens

M. J.

(2015). Five ways to look at Cohen’s kappa. Journal of Psychology & Psychotherapy, 5(4), 197. https://doi.org/10.4172/2161-0487.1000197

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.12 MB

0.00 MB