Short-cut estimators of criterion-referenced test consistency

Abstract

The purpose of this paper is to present relatively easy-to-calculate estimates for the consistency of criterion-referenced tests (CRTs). Four techniques are demonstrated using data from a CRT administered in the intermediate ESL reading classes in the English Language Institute at the University of Hawaii at Manoa. First, the threshold loss agreement approach is represented by the agreement coefficient (Subkoviak 1980) and kappa coefficient (Cohen 1960). Until recently, these coefficients could only be applied in test-retest CRT situa tions. Alternative and simpler methods (suggested by Subkoviak 1988) are given for estimating these coefficients from the results of a single CRT administration. Second, the squared-error loss agreement approach is repre sented by the phi(lambda) dependability index, for which simplified calcula tions, derived from Brennan (1980), are given. Finally, the domain score dependability approach is represented by a short-cut method for estimating a general purpose phi coefficient. This short-cut estimate is derived from Brennan's 1983 and 1984 discussions of the phi coefficient.