Answer-until-correct (AUC) tests have been in use for some time. Pressey (1950) pointed to their ad vantages in enhancing learning, and Brown (1965) proposed a scoring procedure for AUC tests that appears to increase reliability (Gilman & Ferry, 1972; Hanna, 1975). This paper describes a new scoring procedure for AUC tests that (1) makes it possible to determine whether guessing is at ran dom, (2) gives a measure of how "far away" guess ing is from being random, (3) corrects observed test scores for partial information, and (4) yields a mea sure of how well an item reveals whether an ex aminee knows or does not know the correct re sponse. In addition, the paper derives the optimal linear estimate (under squared-error loss) of true score that is corrected for partial information, as well as another formula score under the assumption that the Dirichlet-multinomial model holds. Once certain parameters are estimated, the latter formula score makes it possible to correct for partial infor mation using only the examinee's usual number- correct observed score. The importance of this for mula score is discussed. Finally, various statistical techniques are described that can be used to check the assumptions underlying the proposed scoring procedure.
Get full access to this article
View all access options for this article.
References
1.
Algina, J., & Noe, M.G.A study of the accuracy of Subkoviak's single-administration estimate of the coefficient of agreement using two true-score estimates. Journal of Educational Measurement, 1978, 15, 101-110.
2.
Baker, F.B., & Hubert, L.J.Inference procedures for ordering theory. Journal of Educational Statistics, 1977,2,217-233.
3.
Barlow, R.E., Bartholomew, D.J., Bremner, J.M., & Brunk, H.D.Statistical inference under order restrictions. New York : Wiley, 1972.
4.
Bliss, L.B.A test of Lord's assumption regarding examinee guessing behavior on multiple-choice tests using elementary school students. Journal of Educational Measurement, 1980,17, 147-153.
5.
Brown, J.Multiple response evaluation of discrimination. The British Journal of Mathematical and Statistical Psychology, 1965,18, 125-137.
6.
Brown, J.C., & Burton, R.R.Diagnostic models in basic mathematical skills, In National Institute of Education, Testing, Teaching, and Learning, Report of a Conference on Research on Testing. Washington DC: U.S. Department of Health, Education and Welfare, 1979.
7.
Cohen, J.A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 1960, 20, 37-46.
8.
Coombs, C.H., Milholland, J.E., & Womer, F.B.The assessment of partial knowledge. Educational and Psychological Measurement, 1956, 16, 13-37.
9.
Cross, L.H., & Frary, R.B.An empirical test of Lord's theoretical results regarding formula scoring of multiple-choice tests. Journal of Educational Measurement , 1977,14, 313-321.
10.
Dalton, H.The measurement of the inequality of incomes. Economic Journal , 1920, 30, 348-361.
11.
Dayton, C.M., & Macready, G.B.A probabilistic model for validation of behavioral hierarchies. Psychometrika, 1976, 41, 189-204.
12.
Duncan, G.T.An empirical Bayes approach to scoring multiple-choice tests in the misinformation model. Journal of the American Statistical Association, 1974,69, 50-57.
13.
Frary, R.B.The effect of misinformation, partial information, and guessing on expected multiple-choice test item scores. Applied Psychological Measurement , 1980, 4, 79-90.
14.
Gibbons, J., Olkin, I., & Sobel, M.Selecting and ordering populations: A new statistical methodology. New York: Wiley, 1977.
15.
Gibbons, J.D., Olkin, I., & Sobel, M.A subset selection technique for scoring items on a multiple choice test. Psychometrika, 1979, 44, 259-270.
16.
Gilman, D.A., & Ferry, P.Increasing test reliability through self-scoring procedures. Journal of Educational Measurement, 1972, 9, 205-207.
17.
Griffin, B.S., & Krutchkoff, R.G.Optimal linear estimators : An empirical Bayes version with application to the binomial distribution. Biometrika , 1971, 58, 195-201.
18.
Hambleton, R.K., Swaminathan, H., Algina, J., & Coulson, D.Criterion-referenced testing and measurement : A review of technical issues and developments. Review of Educational Research, 1978, 48, 1-47.
19.
Hanna, G.S.Incremental reliability and validity of multiple-choice tests with an answer-until-correct procedure. Journal of Educational Measurement, 1975,12, 175-178.
20.
Harris, C.W., & Pearlman, A.P.An index for a domain of completion or short answer items. Journal of Educational Statistics, 1978, 3, 285-304.
21.
Hartke, A.R.The use of latent partition analysis to identify homogeneity of an item population . Journal of Educational Measurement, 1978 , 15, 43-47.
22.
Hays, W.Statistics for the social sciences. New York: Holt, Rinehart & Winston, 1973.
23.
Hildebrand, D.K., Laing, J.D., & Rosenthal, H.Prediction analysis of cross classifications. New York : Wiley, 1977.
24.
Hilke, R., Kempf, W.F., & Scandura, J.M.Deterministic and probabilistic theorizing in structural learning. In H. Spada & F. Kempf (Eds.), Structural models of thinking and learning. Bern: Huber, 1977.
25.
Horst, P.The difficulty of a multiple choice test item. Journal of Educational Psychology, 1933, 24, 229-232.
Lord, F.M., & Novick, M.R.Statistical theories of mental test scores. Reading MA : Addison-Wesley, 1974.
32.
Lorenz, M.O.Methods of measuring concentration of wealth. Journal of the American Statistical Association, 1905, 9, 209-219.
33.
Macready, G.B., & Dayton, C.M.The use of probabilistic models in the assessment of mastery. Journal of Educational Statistics, 1977,2, 99-120.
34.
Marshall, A.W., & Olkin, I.Inequalities: Theory of majorization and its applications. New York: Academic Press, 1979.
35.
Morrison, D.G., & Brockway, G.A modified beta-binomial model with applications to multiple choice and taste tests. Psychometrika, 1979, 44, 427-442.
36.
Mosimann, J.E.On the compound multinomial distribution, the multivariate β-distribution, and correlations among proportions. Biometrika, 1962, 49, 65-82.
37.
Pigou, A.C.Wealth and welfare. New York: Macmillan , 1912.
38.
Pressey, S.L.Development and appraisal of devices providing immediate automatic scoring of objective tests and concomitant self-instruction. The Journal of Psychology, 1950, 29, 419-447.
39.
Reulecke, W.A.A statistical analysis of deterministic theories. In H. Spada & F. Kempf (Eds.), Structural models of thinking and learning. Bern: Huber, 1977.
40.
Robertson, T.Testing for and against an order restriction on multinomial parameters. Journal of the American Statistical Association, 1978, 73, 197-202.
41.
Scandura, J.M.Deterministic theorizing in structural learning. Journal of Structural Learning, 1971, 3, 21-53.
42.
Scandura, J.M.Structural learning: Theory and research. New York: Gordon & Breach, 1973.
43.
Spada, H.Logistic models of learning and thought. In H. Spada & F. Kempf (Eds.), Structural models of thinking and learning. Bern: Huber, 1977.
44.
Subkoviak, M.Empirical investigation of procedures for estimating reliability for mastery tests. Journal of Educational Measurement, 1978, 15, 111-116.
45.
Weitzman, R.A.Ideal multiple-choice items. Journal of the American Statistical Association, 1970, 65, 71-89.
46.
Wilcox, R.R.New methods for studying stability. In C. W. Harris, A. Pearlman, & R. Wilcox, Achievement test items—Methods of study (CSE Monograph No. 6. Los Angeles: University of California, Center for the Study of Evaluation, 1977. (a)
47.
Wilcox, R.R.Estimating the likelihood of false-positive and false-negative decisions in mastery testing : An empirical Bayes approach. Journal of Educational Statistics, 1977, 2, 289-307. (b)
48.
Wilcox, R.R.Estimating true score in the compound binomial error model. Psychometrika, 1978, 43, 245-258.
49.
Wilcox, R.R.Achievement tests and latent structure models. British Journal of Mathematical and Statistical Psychology, 1979, 32, 61-71.
50.
Wilcox, R.R.Determining the length of a criterion-referenced test. Applied Psychological Measurement, 1980, 4, 425-446.
51.
Wilcox, R.R.A review of the beta-binomial model and its extensions. Journal of Educational Statistics, in press. (a)
52.
Wilcox, R.R.Analyzing the distractors of multiple-choice test items or partitioning multinomial cell probabilities with respect to a standard. Educational and Psychological Measurement, in press. (b)
53.
Wilcox, R.R.The single administration estimate of the proportion of agreement of a proficiency test scored with a latent structure model. Educational and Psychological Measurement, in press. (c)
54.
Wilks, S.S.Mathematical statistics. New York: Wiley, 1962.
55.
Zehna, P.W.Invariance of maximum likelihood estimation. Annals of Mathematical Statistics, 1966, 37, 744.
