Group Dependence of Some Reliability Indices for Mastery Tests

Abstract

Reliability indices for mastery tests depend not only on true-score variance but also on mean and cutoff scores. This dependence was examined in the case of three decision-theoretic indices: (1) the co efficient of agreement; (2) kappa; and (3) the pro portion of correct decisions. The binomial error model was assumed, with a two-parameter beta dis tribution for true scores. The reliability indices were computed at five values of the mean, four values of KR-21, and four cutoff scores. Results show that the dependence of kappa on mean and cutoff scores is opposite to that of the proportion of correct de cisions, which is linearly related to average thresh old loss. Moreover, kappa can be very small when most examinees are classified correctly. Thus, ob jections against the classical reliability coefficient apply even more strongly to kappa.