For a test that consists of dichotomously scored items, several approaches have been reported in the literature for estimating classification consistency and accuracy indices based on a single administration of a test. Classification consistency and accuracy have not been studied much, however, for ``complex'' assessments—for example, those that involve polytomously scored items or mixtures of different types of items. This article describes procedures for estimating various types of single-administration classification consistency and accuracy indices using the multinomial and compound multinomial models for individuals and groups who take a single administration of a complex assessment. An alternative procedure using the bootstrap is also proposed. These procedures are illustrated using real data sets obtained from tests consisting of both polytomous and dichotomous items.
American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (1999). Standards for educational and psychological testing . Washington, DC: Author.
2.
Brennan, R.L. (2001). Generalizability theory. New York: Springer-Verlag.
3.
Brennan, R.L. (2004). BB-CLASS: A computer program that uses the beta-binomial model for classification consistency and accuracy (Version 1.0, CASMA Research Report No. 9). Iowa City: University of Iowa, Center for Advanced Studies in Measurement and Assessment. Available at http://www.education.uiowa.edu/casma
4.
Brennan, R.L., Harris, D.J., & Hanson, B.A. (1987). The bootstrap and other procedures for examining the variability of estimated variance components in testing contexts (American College Testing Research Report No. 87-7). Iowa City, IA: ACT, Inc.
5.
Brennan, R.L., & Lee, W. (2006a). Correcting for bias in single-administration decision consistency indexes (CASMA Research Report No. 18). Iowa City: University of Iowa, Center for Advanced Studies in Measurement and Assessment. Available at http://www.education.uiowa.edu/casma
6.
Brennan, R.L., & Lee, W. (2006b). Some perspectives on KR-21 (CASMA Technical Note No. 2). Iowa City: University of Iowa, Center for Advanced Studies in Measurement and Assessment. Available at http://www.education.uiowa.edu/casma
7.
Breyer, F.J., & Lewis, C. (1994). Pass-fail reliability for tests with cut scores: A simplified method (ETS Research Report No. 94-39). Princeton, NJ: Educational Testing Service.
8.
Efron, B. (1982). The jackknife, the bootstrap, and other resampling plans. Philadelphia: Society for Industrial and Applied Mathematics.
9.
Efron, B., & Tibshirani, R. (1986). Bootstrap methods for standard errors, confidence intervals, and other measures of statistical accuracy. Statistical Science, 1, 54-77.
10.
Hanson, B.A., & Brennan, R.L. (1990). An investigation of classification consistency indexes estimated under alternative strong true score models. Journal of Educational Measurement, 27, 345-359.
11.
Huynh, H. (1976). On the reliability of decisions in domain-referenced testing. Journal of Educational Measurement, 13, 253-264.
12.
Kelley, T.L. (1947). Fundamentals of statistics. Cambridge, UK: Harvard University Press.
13.
Lee, W. (2007). Multinomial and compound multinomial error models for tests with complex item scoring. Applied Psychological Measurement, 31, 255-274.
14.
Lee, W. (2008). MULT-CLASS: A computer program for multinomial and compound-multinomial classification consistency and accuracy (Version 3.0). Iowa City: University of Iowa, Center for Advanced Studies in Measurement and Assessment. Available at http://www.education.uiowa.edu/casma
15.
Lee, W., Hanson, B.A., & Brennan, R.L. (2002). Estimating consistency and accuracy indices for multiple classifications. Applied Psychological Measurement , 26, 412-432.
16.
Livingston, S.A., & Lewis, C. (1995). Estimating the consistency and accuracy of classifications based on test scores. Journal of Educational Measurement, 32, 179-197.
17.
No Child Left Behind Act of 2001, Pub. L. No. 107-110, 115 Stat. 1425 (2002).
18.
Peng, C.J., & Subkoviak, M.J. (1980). A note on Huynh's normal approximation procedure for estimating criterion-referenced reliability. Journal of Educational Measurement, 17, 359-368.
19.
Subkoviak, M.J. (1976). Estimating reliability from a single administration of a criterion-referenced test. Journal of Educational Measurement , 13, 265-276.
20.
Wan, L., Brennan, R.L., & Lee, W. (2007). Estimating classification consistency for complex assessments (CASMA Research Report No. 22). Iowa City: University of Iowa, Center for Advanced Studies in Measurement and Assessment . Available at http://www.education.uiowa.edu/casma
21.
Woodruff, D.J., & Sawyer, R.L. (1989). Estimating measures of pass-fail reliability from parallel half-tests. Applied Psychological Measurement, 13, 33-43.
