There are many reasons to believe that open-ended (OE) and multiple-choice (MC) items elicit different cognitive demands of students. However, empirical evidence that supports this view is lacking. In this study, we investigated the reactions of test takers to an interactive assessment with immediate feedback and answer-revision opportunities for the two types of items. Eighth-grade students solved mathematics problems, both MC and OE, with standard instructions and feedback-and-revision opportunities. An analysis of scores based on revised answers in feedback mode revealed gains in measurement precision for OE items but not for MC items. These results are explained through the concept of effortful engagement—the OE format encourages more mindful engagement with the items in interactive mode. This interpretation is supported by analyses of response times and test takers’ reports.
AttaliY. (2015a). Effects of multiple-try feedback and question type during mathematics problem solving on performance in similar problems. Computers and Education, 86, 260-267.
2.
AttaliY. (2015b). Practicing for a quantitative reasoning assessment: Effects of question and feedback type. Manuscript submitted for publication.
3.
AttaliY.PowersD. (2010). Immediate feedback and opportunity to revise answers to open-ended questions. Educational and Psychological Measurement, 70, 22-35.
4.
Bangert-DrownsR. L.KulikJ. A.KulikC.-L. C.MorganM. T. (1991). The instructional effect of feedback in test-like events. Review of Educational Research, 61, 213-238.
5.
BaumertJ.DemmrichA. (2001). Testing motivation in the assessment of student skills: The effects of incentives on motivation and performance. European Journal of Psychology of Education, 16, 441-462.
BennettR. E.RockD. A.WangM. (1991). Equivalence of free-response and multiple choice items. Journal of Educational Measurement, 28, 77-92.
8.
BennettR. E.WardW. C. (1993). Construction versus choice in cognitive measurement: Issues in constructed response, performance testing, and portfolio assessment. Hillsdale, NJ: Lawrence Erlbaum.
9.
BlackP.WiliamD. (1998). Inside the black box: Raising standards through classroom assessment. Phi Delta Kappan, 80, 139-148.
10.
BloomB. S. (1984). The 2 sigma problem: The search for methods of group instruction as effective as one-to-one tutoring. Educational Researcher, 13, 4-16.
11.
BraunH.KirschI.YamamotoK. (2011). An experimental study of the effects of monetary incentives on performance on the 12th-grade NAEP reading assessment. Teachers College Record, 113, 2309-2344.
12.
BridgemanB. (1992). A comparison of quantitative questions in open-ended and multiple-choice formats. Journal of Educational Measurement, 29, 253-271.
13.
EklofH. (2006). Development and validation of scores from an instrument measuring test-taking motivation. Educational and Psychological Measurement, 66, 643-656.
14.
EpsteinM. L.EpsteinB. B.BrosvicG. M. (2001). Immediate feedback during academic testing. Psychological Reports, 88, 889-894.
15.
FeldtL. S. (1969). A test of the hypothesis that Cronbach’s alpha or Kuder-Richardson coefficient twenty is the same for two tests. Psychometrika, 34, 363-373.
16.
FeldtL. S. (1980). A test of the hypothesis that Cronbach’s alpha reliability coefficient is the same for two tests administered to the same sample. Psychometrika, 45, 99-105.
17.
GilmanD.FerryP. (1972). Increasing test reliability through self-scoring procedures. Journal of Educational Measurement, 9, 205-207.
18.
HattieJ.TimperleyH. (2007). The power of feedback. Review of Educational Research, 77, 81-112.
19.
KlugerA. N.DeNisiA. (1996). The effects of feedback interventions on performance: A historical review, a meta-analysis, and a preliminary feedback intervention theory. Psychological Bulletin, 119, 254-284.
20.
LaitusisC.AttaliY.StoneE. (2015). Effects of feedback and revision for students with learning disabilities. Manuscript in preparation.
21.
MartinezM. E. (1999). Cognition and the question of test item format. Educational Psychologist, 34, 207-218.
O’NeilH. F.Jr.SugrueB.BakerE. (1996). Effects of motivational interventions on the NAEP mathematics performance. Educational Assessment, 3, 135-157.
25.
PresseyS. L. (1926). A simple apparatus which gives tests and scores—and teaches. School and Society, 23, 373-376.
26.
PresseyS. L. (1950). Development and appraisal of devices providing immediate automatic scoring of objective tests and concomitant self-instruction. Journal of Psychology, 29, 417-447.
27.
RodriguezM. C. (2003). Construct equivalence of multiple-choice and constructed-response items: A random effects synthesis of correlations. Journal of Educational Measurement, 40, 163-184.
28.
RodriguezM. C. (2004). The role of classroom assessment in student performance on TIMSS. Applied Measurement in Education, 17, 1-24.
29.
SalomonG.GlobersonT. (1987). Skill may not be enough: The role of mindfulness in learning and transfer. International Journal of Educational Research, 11, 623-637.
30.
ScrivenM. (1967). The methodology of evaluation. In TylerR. W.GagneR. M.ScrivenM. (Eds.), Perspectives of curriculum evaluation (pp. 39-83). Chicago, IL: Rand McNally.
31.
ShuteV. J. (2008). Focus on formative feedback. Review of Educational Research, 78, 153-189.
32.
SnowR. E.LohmanD. F. (1989). Implications of cognitive psychology for educational measurement. In LinnR. L. (Ed.), Educational measurement (3rd ed., pp. 263-331). New York, NY: Macmillan.
33.
TraubR. E. (1993). On the equivalence of the traits assessed by multiple-choice and constructed-response tests. In BennettR. E.WardW. C. (Eds.), Construction versus choice in educational measurement: Issues in constructed response, performance testing, and portfolio assessment (pp. 29-43). Hillsdale, NJ: Lawrence Erlbaum.
34.
TraubR. E.MacRuryK. (1990). Multiple-choice vs. free-response in the testing of scholastic achievement. In IngenkampK.JagerR. S. (Eds.), Yearbook on educational measurement (pp. 128-159). Weinheim, Germany: Beltz.
35.
WainerH. (1993). Measurement problems. Journal of Educational Measurement, 30, 1-21.
36.
WainerH.ThissenD. (1993). Combining multiple-choice and constructed-response test scores: Toward a Marxist theory of test construction. Applied Measurement in Education, 6, 103-118.
37.
WardW. C.DupreeD.CarlsonS. B. (1987). A comparison of free-response and multiple-choice questions in the assessment of reading comprehension (RR-87-20). Princeton, NJ: Educational Testing Service.
38.
WiliamD.ThompsonM. (2008). Integrating assessment with learning: What will it take to make it work? In DwyerC. A. (Ed.), The future of assessment: Shaping teaching and learning (pp. 53-82). New York, NY: Lawrence Erlbaum.
39.
WiseS. L.DeMarsC. E. (2005). Low examinee effort in low-stakes assessment: Problems and potential solutions. Educational Assessment, 10, 1-17.
