Consider an n-item multiple-choice test where it is decided that an examinee knows the answer if and only if he/she gives the correct response. The k out of n reliability of the test, Qk, is defined to be the probability that, for a randomly sampled examinee, at least k correct decisions are made about whether the examinee knows the answer to an item. The paper describes and illustrates how an extension of a recently proposed latent structure model can be used in conjunction with results in Sathe, Pradhan, and Shah (1980) to estimate upper and lower bounds on Qk. A method of empirically checking the model is discussed.
Get full access to this article
View all access options for this article.
References
1.
Barlow, R.E., Bartholomew, D.J., Bremner, J.M., & Brunk, H.D.Statistical inference under order restrictions. New York : Wiley, 1972.
2.
Bergan, J.R., Cancelli, A.A., & Luiten, J.W.Mastery assessment with latent class and quasi-independence models representing homogeneous item domains. Journal of Educational Statistics , 1980, 5, 65-81.
3.
Bliss, L.B.A test of Lord's assumption regarding examinee guessing behavior on multiple-choice tests using elementary school students. Journal of Educational Measurement, 1980,17, 147-153.
4.
Brownless, V.T., & Keats, J.A.A retest method of studying partial knowledge and other factors influencing item response. Psychometrika, 1958, 23, 67-73.
5.
Cross, L.H., & Frary, R.B.An empirical test of Lord's theoretical results regarding formula-scoring of multiple-choice tests. Journal of Educational Measurement , 1977,14, 313-321.
6.
Dayton, C.M., & Macready, G.B.A probabilistic model for validation of behavioral hierarchies. Psychometrika, 1976, 41, 189-204.
7.
Dayton, C.M., & Macready, G.B.A scaling model with response errors and intrinsically unscalable respondents. Psychometrika, 1980 , 45, 343-356.
8.
Katti, S.K.Exact distribution for the chi-square test in the one-way table. Communications in Statistics, 1973, 2, 435-447.
9.
Knapp, T.R.The reliability of a dichotomous test-item: A 'correlationless' approach. Journal of Educational Measurement, 1977, 14, 237-252.
10.
Macready, G.B., & Dayton, C.M.The use of probabilistic models in the assessment of mastery. Journal of Educational Statistics, 1977, 2, 99-120.
11.
Macready, G.B., & Dayton, C.M.The nature and use of state mastery models. Applied Psychological Measurement, 1980, 4, 493-516.
12.
Marks, E., & Noll, G.A.Procedures and criteria for evaluating reading and listening comprehension tests. Educational and Psychological Measurement, 1967, 27, 335-348.
13.
Molenaar, I.On Wilcox's latent structure model for guessing. British Journal of Mathematical and Statistical Psychology, 1981, 34, 224-228.
14.
Pearson, K.Tables of the incomplete beta function. Cambridge, England: Cambridge University Press, 1968.
15.
Robertson, T.Testing for and against an order restriction on multinomial parameters. Journal of theAmerican Statistical Association, 1978 , 73, 197-202.
16.
Sathe, Y.S., Pradhan, M., & Shah, S.P.Inequalities for the probability of the occurrence of at least m out of n events. Journal of Applied Probability, 1980 , 17, 1127-1132.
17.
Smith, P.J., Rae, D.S., Manderscheid, R.W., & Silbergeld, S.Exact and approximate distributions of the chi-square statistic for equiprobability, Communications in Statistics—Simulation and Computation, 1979, B8, 131-149.
18.
White, R.T., & Clark, R.M.A test of inclusion which allows for errors of measurement. Psychometrika, 1973, 38, 77-86.
19.
Wilcox, R.R.New methods for studying stability. In C. W. Harris, A. Pearlman, & R. Wilcox, Achievement test items: Methods of study (CSE Monograph No. 6). Los Angeles: University of California, Center for the Study of Evaluation, 1977. (a)
20.
Wilcox, R.R.New methods for studying equivalence. In C. W. Harris, A. Pearlman, & R. Wilcox, Achievement test items: Methods of study (CSE Monograph No. 6). Los Angeles: University of California, Center for the Study of Evaluation, 1977. (b)
21.
Wilcox, R.R.Solving measurement problems with an answer-until-correct scoring procedure . Applied Psychological Measurement, 1951, 5, 399-414. (a)
22.
Wilcox, R.R.Recent advances in measuring achievement: A response to Molenaar. British Journal of Mathematical and Statistical Psychology, 1981, 34, 229-237. (b)
23.
Wilcox, R.R.Some new results on an answer-until-correct scoring procedure. Journal of Educational Measurement, 1982, 19, 67-74.
24.
Wilcox, R.R.Some empirical and theoretical results on an answer-until-correct scoring procedure. British Journal of Mathematical and Statistical Psychology, in press. (a)
25.
Wilcox, R.R.Using results on k out of n system reliability to study and characterize tests . Educational and Psychological Measurement, in press. (b)
