Ability estimates obtained from applying finite state score theory, item response models, and classical test theory to score multiple-choice items were compared and contrasted. Responses of 1,230 subjects to the multiple-choice portion of a school-leaving reading comprehension examination were used. Correlations among the ability estimates were nearly perfect. Furthermore, the mean absolute difference values between finite state scores and ability estimates based on the one- and two-parameter models were smaller than that between finite state scores and the ability estimates based on the three-parameter model. The mean absolute difference value between finite state scores and classical test scores was the largest. However, whereas the item response theory and classical test score theory analyses provide item-level information, that is not the case for finite state score theory.
Get full access to this article
View all access options for this article.
References
1.
Aiken, L. R. (1987). Testing with multiple-choice items. Journal of Research and Development in Education, 20(4), 44-58.
2.
Assessment System Corporation . (1989). User's manual for ASCAL: 2- and 3-parameter IRT calibration program. St. Paul, MN: Author.
3.
Assessment System Corporation . (1992). User's manual for RASCAL: Rasch analysis program. St. Paul, MN: Author.
4.
Assessment System Corporation . (1993). User's manual for ITEMAN: Conventional item analysis program. St. Paul, MN: Author.
5.
Bennet, R. E. , & Ward, W. C. (1993). Construction versus choice in cognitive measurement: Issues in constructed response, performance testing, and portfolio assessment. Hillsdale, NJ: Lawrence Erlbaum.
6.
Bock, R. D. (1972). Estimating item parameters and latent ability when responses are scored in two or more nominal categories. Psychometrika, 37, 29-51.
7.
Claudy, J. G. (1978). Biserial weights: A new approach to test item option weighting. Applied Psychological Measurement, 2, 25-30.
8.
Frary, R. B. (1989). Partial-credit scoring methods for multiple-choice tests: Comparative review. Applied Measurement in Education, 2(1), 79-96.
9.
Garcia-Perez, M. A. (1987). A finite state theory of performance in multiple-choice tests. In E. Roskam & R. Suck (Eds.), Progress in mathematical psychology (Vol. 1, pp. 455-464). Amsterdam: Elsevier.
10.
Garcfa-Perez, M. A. (1993). In defence of none of the above. British Journal of Mathematical and Statistical Psychology, 46, 213-229.
11.
Garcia-Perez, M. A. (1994). Parameter estimation and goodness-of-fit testing in multinomial models. British Journal of Mathematical and Statistical Psychology, 47, 247-282.
12.
García-Pérez, M. A. , & Frary, R. B. (1989). Psychometric properties of finite state scores versus number-correct and formula scores: A simulation study. Applied Psychological Measurement, 13, 403-417.
13.
Garcfa-Perez, M. A. , & Frary, R. B. (1991). Finite state polynomic item characteristic curves. British Journal of Mathematical and Statistical Psychology, 44, 45-75.
14.
Glaser, R. (1981). The future of testing: A research agenda for cognitive psychology and psychometrics. American Psychologist, 36, 923-936.
15.
Glass, G. V. , & Hopkins, K. D. (1984). Statisticat methods in education and psychology. Boston: Allyn & Bacon.
16.
Haladyana, T. M. (1994). Developing and validating multiple-choice test items. Hillsdale, NJ: Lawrence Erlbaum.
17.
Hutchinson, T. P. (1985, July). Predicting performance in variants of the multiple-choice test. Paper presented at the annual meeting of the Psychometric Society and the Classification Societies, Cambridge, England. (ERIC Document No. ED 263 177)
18.
Maguire, T. O. , Hattie, J., & Haig, B. (1994). Construct validity and achievement assessment. Alberta Journal of Educational Research, 40(2), 109-126.
19.
Masters, G. N. (1982). A Rasch model for partial credit scoring. Psychometrika, 47(2), 149-173.
20.
Rogers, W. T. , & Bateson, D. J. (1991). Verification of a model of test-taking behaviour of high school seniors. Journal of Experimental Education, 59, 331-350.
21.
Rogers, W. T., & Wilson, C. (1993). The influence of test-wiseness upon performance of high school students on Alberta Diploma Examinations. Unpublished research report, Centre for Research in Applied Measurement and Evaluation, University of Alberta.
22.
Samejima, F. (1969). Estimation of latent ability using a response pattern of graded scores. Psychometrika Monograph Supplement, No. 17.
23.
Smith, R. M. (1987). Assessing partial knowledge in vocabulary. Journal of Educational Measurement, 24, 217-233.
24.
Suen, H. K. (1990). Principles of test theories. Hillsdale, NJ: Lawrence Erlbaum.
25.
Thissen, D. , Steinberg, L., & Fitzpatrick, A. R. (1989). Multiple-choice models: The distractors are also part of the item. Journal of Educational Measurement, 26, 161-177.
26.
Wainer, H. (1989). The future of item analysis. Journal of Educational Measurement, 26, 191-208.
27.
Zin, T. T. (1992). Comparing 12 finite state models of examinee performance on multiple-choice tests. Dissertation Abstracts International, 53(4), 1137. (Doctoral dissertation, Virginia Polytechnic and State University)
