The present study investigated the effects of reducing the number of options per item
on psychometric characteristics of a Japanese EFL university entrance examination. A
four-option multiple-choice reading test used for entrance screening at a university
in Japan was later converted to a three-option version by eliminating the least
frequently endorsed option in each item, and was given to a separate group.
Responses to the two tests indicated that using three options instead of four did
not significantly change the mean item facility or the mean item discrimination.
Distractor analyses revealed that whether four or three options were provided, the
actual test-takers’ responses spread, on the average, over about 2.6
options per item, that the mean number of functioning distractors was much lower
than 2, and that reducing the least popular option had only a minimal effect on the
performance of the remaining options. These results suggested that three-option
items performed nearly as well as their four-option counterparts.
Get full access to this article
View all access options for this article.
References
1.
Aamodt, M. G.
and
McShane, T.1992:
A meta-analytic investigation of the effect of various test item
characteristics on test scores and test completion times
. Public Personnel Management21,
151-160
.
2.
Brown, J. D.1995:
English language entrance examinations in Japan: myths and facts
. The Language Teacher19,
21-26
.
3.
Bruno, J. E.
and
Dirkzwager, A.1995:
Determining the optimal number of alternatives to a multiple-choice test
item: an information theoretic perspective
. Educational and Psychological Measurement55,
959-966
.
4.
Budescu, D. V.
and
Nevo, B.1985:
Optimal number of options: an investigation of the assumption of proportionality
. Journal of Educational Measurement22,
183-196
.
5.
Cizek, G. J.
and
O’Day, D. M.1994:
Further investigation of nonfunctioning options in multiple-choice test items
. Educational and Psychological Measurement54,
861-872
.
6.
Costin, F.1970:
The optimal number of alternatives in multiple-choice tests: some empirical
evidence for a mathematical proof
. Educational and Psychological Measurement30,
353-358
.
7.
Costin, F.1972:
Three-choice versus four-choice items: implications for reliability and
validity of objective achievement tests
. Educational and Psychological Measurement32,
1035-1038
.
8.
Crehan, K. D.
,
Haladyna, T. M.
and Brewer, B.W. 1993:
Use of an inclusive option and the optimal number of options for
multiple-choice items
. Educational and Psychological Measurement53,
241-247
.
9.
Delgado, A. R.
and
Prieto, G.1998:
Further evidence favoring three-option items in multiple-choice tests
. European Journal of Psychological Assessment14,
197-201
.
10.
Downing, S. M.1992:
True-false and alternative-choice formats: a review of research
. Educational Measurement: Issues and Practices11,
27-30
.
11.
Ebel, R. L.1968:
Blind guessing on objective achievement tests
. Journal of Educational Measurement5,
321-325
.
12.
Green, K.
,
Sax, G.
and
Michael, W. B.1982:
Validity and reliability of tests having different numbers of options for
students of differing levels of ability
. Educational and Psychological Measurement42,
239-245
.
13.
Gronlund, N.1988: How to construct achievement tests. 4th
Edition.
Prentice-Hall
.
14.
Haladyna, T. M.1997: Writing test items to evaluate higher order thinking.
Allyn and Bacon
.
15.
Haladyna, T. M.1999: Developing and validating multiple-choice test items.
2nd Edition.
Lawrence Erlbaum
.
16.
Haladyna, T. M.
and
Downing, S. M.1989:
Validity of a taxonomy of multiple-choice item-writing rules
. Applied Measurement in Education2,
51-78
.
17.
Haladyna, T. M.
and
Downing, S. M.1993:
How many options is enough for a multiple-choice test items
? Educational and Psychological Measurement53,
999-1010
.
18.
Haladyna, T. M.
,
Downing, S. M.
and
Rodriguez, M. C.2002:
A review of multiple-choice item-writing guidelines for classroom assessment
. Applied Measurement in Education15,
309-334
.
19.
Landrum, R. E.
,
Cashin, J. R.
and
Theis, K. S.1993:
More evidence in favor of three-option multiple-choice tests
. Educational and Psychological Measurement53,
771-778
.
20.
Linacre, J. M.2004: A user’s guide to WINSTEPS/MINISTEPS: Rasch-model
computer programs.
Institute for Objective Measurement
.
21.
Linn, R. L.
and
Gronlund, N. E.2000: Measurement and assessment in teaching. 8th
Edition.
Prentice-Hall
.
22.
Lord, F.1977:
Optimal number of choices per item: a comparison of four approaches
. Journal of Educational Measurement14,
33-38
.
23.
Owen, S. V.
and
Froman, R. D.1987:
What’s wrong with three-option multiple-choice items
? Educational and Psychological Measurement47,
513-522
.
24.
Ramos, R. J.
and
Stern, J.1973:
Item behavior associated with changes in the number of alternatives in
multiple choice items
. Journal of Educational Measurement10,
305-310
.
25.
Rogers, W. T.
and
Harley, D.1999:
An empirical comparison of three- and four- choice items and tests:
susceptibility to testwiseness and internal consistency reliability
. Educational and Psychological Measurement59,
234-247
.
26.
Sato, T.
and
Morimoto, U.
1976:
Sentaku-shi keishiki tesuto kaitou bunpu no bunseki [Analyzing endorsement
distribution of selected-response items]
. In Proceedings of the 4th Annual Meeting of the Behaviometric
Society of Japan. Behaviometric Society of Japan.
27.
Sidick, J. T.
,
Barrett, G. V.
and
Doverspike, D.1994:
Three-alternative multiple-choice tests: an attractive option
. Personnel Psychology47,
829-835
.
28.
Smith, R. M.
,
Schumacker, R. E.
and
Bush, M. J.1998:
Using item mean squares to evaluate fit to the Rasch model
. Journal of Outcome Measurement2,
66-78
.
29.
Straton, R. G.
and
Catts, R. M.1980:
A comparison of two, three, and four-choice item tests given a fixed total
number of choices
. Educational and Psychological Measurement40,
357-365
.
30.
Trevisan, M. S.
,
Sax, G.
and
Michael, W. B.1991:
The effects of the number of options per item and student ability on test
validity and reliability
. Educational and Psychological Measurement51,
829-837
.
31.
Trevisan, M. S.
,
Sax, G.
and
Michael, W. B.1994:
Estimating the optimum number of options per item using an incremental
option paradigm
. Educational and Psychological Measurement54,
86-91
.
32.
Tversky, A.1964:
On the optimal number of alternatives at a choice point
. Journal of Mathematical Psychology1,
386-391
.
33.
Williams, B. J.
and
Ebel, R. L.1957: The effect of varying the number of alternatives per item
on multiple-choice vocabulary test items. In
Huddleton, E. M.
, editor, The 14th Yearbook of the National Council on Measurement in
Education,
Michigan State University
, 63-65.
34.
Wright, B. D.
and
Masters, G. N.1982: Rating scale analysis.
MESA Press
.
