Empirical Choice Weighting Under "Guess" and "Do Not Guess" Directions

Abstract

This study investigated reliability and validity of scores resulting from empirical choice-weighting under two conditions: (1) examinees were told not to guess when unable to eliminate any choice and (2) they were told their scores would be the number of right responses. Unlike other choice-weighting studies which have reported disappointing results under one or the other of these two conditions, the present study used a predictor and criterion which were obtained independently yet had a substantial logical relationship. The results indicated that choice-weighting resulted in substantial increases in score reliability and predictive validity over number-right and corrected-for-guessing scores. Moreover, the validity coefficients associated with choice-weighted scores obtained under "Guess" directions were comparable to coefficients obtained under "Do Not Guess" directions. This latter result led to a recommendation against instruction to restrict guessing under choice-weighted scoring.

Get full access to this article

View all access options for this article.

References

Ashler, D. Itemrs IV, Unpublished Computer Program available through the Center for Research in Evaluation and Measurement , Graduate School of Education, University of Pennsylvania .

Bayuk, R. J. The effects of choice weights and item weights on the reliability and predictive validity of reading tests . Paper presented at the annual meeting of the American Educational Research Association, Chicago, Illinois, 1974.

Cross, L. H. and Frary, R. B. An empirical test of Lord's Theoretical results regarding formula scoring of multiple-choice tests. Journal of Educational Measurement, 1977, 14, 313-321.

Davis, F. B. A note on the correction for chance success. Journal of Experimental Education, 1967, 35, 43-47.

Davis, F. B. and Fifer, G. The effect on test reliability and validity of scoring aptitude and achievement tests with weights for every choice. EDUCATIONAL AND PSYCHOLOGICAL MEASUREMENT, 1959, 19, 159-170.

Echternacht, G. Reliability and validity of option weighting schemes. EDUCATIONAL AND PSYCHOLOGICAL MEASUREMENT , 1976, 36, 301-309.

Hendrickson, G. F. The effect of differential option weighting on multiple-choice objective tests. Journal of Educational Measurement, 1971, 8, 291-296.

Kansup, W. and Hakstian, R. A. A comparison of several methods of assessing partial knowledge in multiple-choice tests: I. Scoring procedures. Journal of Educational Measurement, 1975, 12, 219-230.

Lord, F. M. Formula scoring and number-right scoring. Journal of Educational Measurement, 1975, 12, 7-11.

10.

Reilly, R. R. Empirical option weighting with a correction-for-guessing. EDUCATIONAL AND PSYCHOLOGICAL MEASUREMENT , 1975, 35, 613-619.

11.

Reilly, R. R. and Jackson, R. Effects of empirical option weighting on reliability and validity of an academic aptitude test. Journal of Educational Measurement, 1973, 10, 185-194.

12.

Sabers, D. L. and White, G. W. The effect of differential weighting on individual item responses on the predictive validity and reliability of an aptitude test. Journal of Educational Measurement, 1969, 6, 93-96.