Many testing programs rely on equating procedures to achieve comparability of scores on alternate test forms. One commonly accepted rule for developing equated examinations using the common-items nonequivalent groups (CINEG) design is that items common to the two examinations being equated should be identical. In test construction practice, this rule has been extended to include even the order in which options appear in the two examinations. The present study examined the performance of a common set of items on an examination in which the order of options for one test form was experimentally manipulated. The study sought to determine whether reordering multiple choice item options results in any significant effect on item difficulty. It was found that reordering options can have significant but unpredictable effects on item performance. A linkage is made to previous research and cautions are suggested regarding the effects of reordering options.
Get full access to this article
View all access options for this article.
References
1.
Ace, M. C. , & Dawis, R. V. (1973). Item structure as a determinant of item difficulty in verbal analogies. Educational and Psychological Measurement, 33, 143-149.
2.
Angoff, W. H. (1971). Scales, norms, and equivalent scores. In R. L. Thormdike (Ed.), Education measurement (2nd ed., pp. 508-600). Washington, DC: American Council on Education.
3.
Benson, J. , & Crocker, L. (1979). The effects of item format and reading ability on objective test performance: A question of validity. Educational and Psychological Measurement, 39, 381-387.
4.
Bolden, B. J. , & Stoddard, A. (1980, April). The effects of language on test performance of elementary school children. Paper presented at the annual meeting of the American Educational Research Association, Boston.
5.
Bornstein, H. , & Chamberlain, K. (1970). An investigation of the effects of "verbal load" in achievement tests. American Educational Research Journal, 7, 597-604.
6.
Braun, H. I. , & Holland, P. W. (1982). Observed score test equating: A mathematical analysis of some ETS equating procedures. In P. W. Holland & D. B. Rubin (Eds.), Test equating (pp. 9-49). New York: Academic Press.
7.
Cassels, J.R.T. , & Johnstone, A. H. (1984). The effect of language on student performance on multiple choice tests in chemistry. Journal of Chemical Education, 61, 613-615.
8.
Dorans, N. J. (1990). Equating methods and sampling designs. Applied Measurement in Education, 3, 3-18.
9.
Ericsson, K. A. , & Simon, H. A. (1984). Protocol analysis. Cambridge: MIT Press.
10.
Green, K. (1984). Effects of item characteristics on multiple-choice item difficulty. Educational and Psychological Measurement, 44, 551-561.
11.
Harris, D. J. (1990, April). Effects of passage and item scrambling on equating relationships. Paper presented at the annual meeting of the American Educational Research Association, Boston.
12.
Jerman, M. E. , & Mirman, S. (1971). Linguistic and computational variables in problem solving in elementary mathematics. Educational Studies in Mathematics, 5, 317-362.
13.
Jessell, J. C. , & Sullins, W. L. (1975). The effect of keyed response sequencing on multiple-choice items on performance and reliability. Journal of Educational Measurement, 12, 45-48.
14.
Klein, L. W. , & Jajoura, D. (1985). The importance of content representativeness for common-item equating with nonrandom groups. Journal of Educational Measurement, 22, 197-206.
15.
Kolen, M. J. (1989). Traditional equating methodology. Educational Measurement: Issues and Practice, 7(4), 29-36.
16.
Kolen, M. J. , & Brennan, R. L. (1987). Linear equating models for the common-item nonequivalent-populations design. Applied Psychological Measurement, 11, 263-277.
17.
Leary, L. F. , & Dorans, N. J. (1985). Implications for altering the context in which items appear: A historical perspective on an immediate concern. Review of Educational Research, 55(3), 387-413.
18.
Loftus, E. F. , & Suppes, P. (1972). Structural variables that determine problem-solving difficulty in computer-assisted instruction. Journal of Educational Psychology, 63, 531-542.
19.
Marcus, A. (1963). The effect of correct response location on the difficulty of multiple-choice questions. Journal of Applied Psychology, 47, 48-51.
20.
McNamara, W. J. , & Weitzman, E. (1945). The effect of choice placement on the difficulty of multiple-choice questions. Journal of Educational Psychology, 36, 103-113.
21.
Millman, J. (1978, July). Determinants of item difficulty: A preliminary investigation (CSE Rep. No. 114). Los Angeles: University of California, Center for the Study of Evaluation.
22.
O'Neill, K. A. (1986, April). The effect of stylistic changes on item performance. Paper presented at the annual meeting of the American Educational Research Association, San Francisco.
23.
Petersen, N. S. , Kolen, M. J., & Hoover, H. D. (1989). Scaling, norming, and equating. In R. L. Linn (Ed.), Educational measurement (3rd ed., pp. 221-262). New York: Macmillan.
24.
Wilbur, P. H. (1966). Positional response set in the multiple-choice examination. Dissertation Abstracts International, 29/09-A, 2902-3051. (University Microfilms No. AAD 67-02, 131)
