The robustness of number-right; one-, two-, and three-parameter item-response; finite-state; and partial-credit scoring was examined with respect to the violation of the equally classifiable options and option independence made in finite-state scoring. All other assumptions underlying the use of these scoring models were met for each of four sub-tests that varied in terms of the violations. Analysis of the responses of 1,232 high school seniors on the subtests revealed that the number-right and one-, two-, and three-parameter scoring methods were equally sensitive to the presence of best answers (lack of option independence) and that the number-right and one- and two-parameter methods were equally sensitive to the presence of absurd option and stem-option connections (unequal classification of options) and pairs of similar or opposite options (lack of option independence, unequal classification of options). The three-parameter model and the finite-state scoring models were adversely sensitive to the presence of testwiseness.
Get full access to this article
View all access options for this article.
References
1.
Aiken, L. R. (1987). Testing with multiple choice items. Journal of Research and Development in Education, 20, 44-58.
2.
Assessment System Corporation . (1993). User’s manual for ITEMAN: Conventional item analysis program. St. Paul, MN: Author.
3.
Bennet, R. E. , & Ward, W. C. (1993). Construction versus choice in cognitive measurement: Issues in constructed response, performance testing, and portfolio assessment. Hillsdale, NJ: Lawrence Erlbaum.
4.
Bock, R. D. (1972). Estimating item parameters and latent ability when responses are scored in two or more nominal categories. Psychometrika, 37, 29-51.
5.
Bruno, J. E. , & Dirkzwager, A. (1995). Determining the optimal number of alternatives to a multiple-choice test item: An information theoretic perspective. Educational and Psychological Measurement, 55, 959-966.
6.
Cattell, R. B. (1952). Factor analysis. New York: Harper & Bros.
7.
Claudy, J. G. (1978). Biserial weights: A new approach to test item option weighting. Applied Psychological Measurement, 2, 25-30.
8.
Costin, F. (1970). The optimal number of alternatives in multiple-choice achievement tests: Some empirical evidence for a mathematical proof. Educational and Psychological Measurement, 30, 353-358.
9.
Costin, F. (1972). Three-choice versus four-choice items: Implications for reliability and validity of objective achievement tests. Educational and Psychological Measurement, 32, 1035-1038.
10.
Fan, X. (1998). Item response theory and classical test theory: An empirical comparison of their item/person statistics. Educational and Psychological Measurement, 58, 357-381.
11.
Frary, R. B. (1988). NCME instructional module: Formula scoring of multiple-choice tests (correction for guessing). Educational Measurement: Issues and Practice, 7(2), 33-39.
12.
Frary, R. B. (1989). Partial-credit scoring methods for multiple choice tests. Applied Measurement in Education, 2, 79-96.
13.
García-Pérez, M. A. (1987). A finite state theory of performance in multiple choice tests. In E. Roskam & R. Suck (Eds.), Progress in mathematical psychology (Vol. 1, pp. 455-464). Amsterdam, the Netherlands: Elsevier.
14.
García-Pérez, M. A. (1994). Parameter estimation and goodness-of-fit testing in multinomial models. British Journal of Mathematical and Statistical Psychology, 47, 247-282.
15.
García-Pérez, M. A. , & Frary, R. B. (1989). Psychometric properties of finite state scores versus number-correct and formula scores: A simulation study. Applied Psychological Measurement, 13, 403-417.
16.
García-Pérez, M. A. , & Frary, R. B. (1991). Finite state polynomic item characteristic curves. British Journal of Mathematical and Statistical Psychology, 44, 45-75.
17.
Glass, G. V , & Hopkins, K. D. (1996). Statistical methods in education and psychology (3rd ed.). Needham Heights, MA: Allyn & Bacon.
18.
Grier, J. B. (1975). The number of alternatives for optimum test reliability. Journal of Educational Measurement, 12, 109-113.
19.
Haladyna, T. M. (1994). Developing and validating multiple-choice test items. Hillsdale, NJ: Lawrence Erlbaum.
20.
Haladyna, T. M. (1997). Writing test items to evaluate higher order thinking. Boston: Allyn & Bacon.
21.
Hambleton, R. K. , & Murphy, E. (1992). A psychometric perspective on authentic measurement. Applied Measurement in Education, 5, 1-16.
22.
Hambleton, R. K , & Murray, L. (1983). Some goodness of fit investigations for item response models. In R. K. Hambleton (Ed.), Applications of item response theory (pp. 71-94). Vancouver: Educational Research Institute of British Columbia.
23.
Hambleton, R. K. , Roberts, D. M., & Traub, R. E. (1970). A comparison of reliability and validity of two methods of assessing partial knowledge on multiple choice tests. Journal of Educational Measurement, 7, 75-82.
24.
Hambleton, R. K. , Swaminathan, H., & Rogers, H. J. (1991). Fundamentals of item response theory. Newbury Park, CA: Sage.
25.
Hutchinson, T. P. (1982). Some theories of performance in multiple choice tests, and their implications for variants of the task. British Journal of Mathematical and Statistical Psychology, 35, 71-89.
26.
Lord, F. M. (1977). Optimal number of choices per item: A comparison of four approaches. Journal of Educational Measurement, 14, 33-38.
27.
Lord, F. M. , & Novick, M. R. (1968). Statistical theories of mental tests scores. Reading, MA: Addison-Wesley.
28.
Maguire, T. O. , Hattie, J., & Haig, B. (1994). Construct validity and achievement assessment. Alberta Journal of Educational Research, 40, 109-126.
29.
Masters, G. N. (1982). A Rasch model for partial credit scoring. Psychometrika, 47, 149-173.
30.
Mislevy, R. J. , & Bock, R. D. (1990). BILOG 3: Item analysis and test scoring with binary logistic models. Chicago: Scientific Software.
31.
Nandakumar, R. (1994). Assessing dimensionality of a set of item responses—Comparison of different approaches. Journal of Educational Measurement, 31, 17-35.
32.
Nandakumar, R. , & Stout, W. F. (1993). Refinements of Stout’s procedure for assessing latent trait dimensionality. Journal of Educational Statistics, 18, 41-68.
33.
Ndalichako, J. , & Rogers, W. T. (1997). Comparison of finite state score theory, classical test theory, and item response theory in scoring multiple-choice items. Educational and Psychological Measurement, 57, 580-589.
34.
Rogers, W. T. , & Bateson, D. J. (1991a). The influence of test-wiseness on performance of high school seniors on school leaving examinations. Applied Measurement in Education, 4, 159-183.
35.
Rogers, W. T. , & Bateson, D. J. (1991b). Verification of a model of test-taking behavior of high school seniors. Journal of Experimental Education, 59, 331-350.
36.
Rogers, W. T. , & Harley, D. (1999). An empirical comparison of three- and four-choice items and tests: Susceptibility to testwiseness and internal consistency reliability. Educational and Psychological Measurement, 59(2), 234-247.
37.
Rogers, W. T. , & Wilson, C. (1993). The influence of test-wiseness upon performance of high school students on Alberta Diploma Examinations. Edmonton: Centre for Research in Applied Measurement and Evaluation, University of Alberta.
38.
Rogers, W. T. , & Yang, P. (1996). Test-wiseness: Its nature and application. European Journal of Psychological Assessment, 12, 247-259.
39.
Rothman, R. (1995). Measuring up: Standards, assessment, and school reform. San Francisco: Jossey-Bass.
40.
Samejima, F. (1969). Estimation of latent ability using a response pattern of graded scores. Psychometrika Monograph Supplement, 17.
41.
Sarnaki, R. E. (1979). An examination of test-wiseness in the cognitive domain. Review of Educational Research, 49, 252-279.
42.
Shepard, L. A. (1991). Interview on assessment issues with Lorri Shepard. Educational Researcher, 20(2), 21-23, 27-27.
43.
Smith, R. M. (1987). Assessing partial knowledge in vocabulary. Journal of Educational Measurement, 24, 217-233.
44.
Stout, W. F. (1987). A non parametric approach for assessing latent trait unidimensionality. Psychometrika, 52, 589-617.
45.
Straton, R. G. , & Catts, R. M. (1980). A comparison of two, three and four-choice item tests given a fixed total number of choices. Educational and Psychological Measurement, 40, 357-365.
46.
Thissen, D. , & Steinberg, L. (1984). A response model for multiple choice items. Psychometrika, 49, 501-519.
47.
Thissen, D. , Steinberg, L., & Fitzpatrick, A. R. (1989). Multiple choice models: The distractors are also part of the item. Journal of Educational Measurement, 26, 161-177.
48.
Tversky, A. (1964). On the optimal number of alternatives at a choice point. Journal of Mathematical Psychology, 1, 386-391.
49.
Wainer, H. (1989). The future of item analysis. Journal of Educational Measurement, 26, 191-208.
50.
Williams, B. J. , & Ebel, R. L. (1957). The effect of varying the number of alternatives per item on multiple-choice vocabulary test items. In E. M. Huddleston (Ed.), Fourteenth yearbook of the National Council on Measurements used in Education (pp. 63-65). Princeton, NJ: NCMUE.
51.
Wright, B. D. , & Linnacre, J. M. (1992). A user’s guide to BIGSTEPS: Rasch-Model Computer Program, Version 2.2. Chicago: MESA.
52.
Zin, T. T. (1992). Comparing 12 finite state models of examinee performance on multiple-choice tests. Dissertation Abstracts International, 53(4), 1137. (Doctoral dissertation, Virginia Polytechnic and State University)
