An Expansion and Practical Evaluation of Expected Classification Accuracy

Abstract

Rudner (2001, 2005) described an expected classification accuracy index for determining the asymptotic expectation of accuracy of classifications of examinees into score categories. This article expands on that exposition by evaluating the index as it is likely to be used in practice (as a point estimate of classification accuracy), provides a related index of expected proportion in each score category, derives a measurement error—based standard error of expected proportion in each category, applies the methods to a large-scale state assessment database, and analyzes the benefits and limitations of the methodology. This study demonstrates that Rudner's index as a point estimate is slightly positively biased but nevertheless useful for groups with reasonably large numbers of examinees. It also reinforces the inappropriateness of estimating proportions in score categories for many small groups.