Computerized testing provides many benefits to support formative assessment. However, the advent of computerized formative testing has also raised formidable new challenges, particularly in the area of item development. Large numbers of diverse, high-quality test items are required because items are continuously administered to students. Hence, hundreds of items are needed to develop the banks necessary for computerized formative testing. One promising approach that may be used to address this test development challenge is automatic item generation. Automatic item generation is a relatively new but rapidly evolving research area where cognitive and psychometric modeling practices are used to produce items with the aid of computer technology. The purpose of this study is to describe a new method for generating both the items and the rationales required to solve the items to produce the required feedback for computerized formative testing. The method for rationale generation is demonstrated and evaluated in the medical education domain.
BejarI. I.LawlessR. R.MorleyM. E.WagnerM. E.BennettR. E.RevueltaJ. (2003). A feasibility study of on-the-fly item generation in adaptive testing. Journal of Technology, Learning, and Assessment, 2(3), 1-32.
2.
BlackP.WiliamD. (1998). Assessment and classroom learning. Assessment in Education: Principles, Policy, & Practice, 5, 7-74.
3.
BlackP.WiliamD. (2010). Inside the black box: Raising standards through classroom assessment. Phi Delta Kappan, 92, 81-90.
4.
Council of Chief State School Officers. (2008). Formative assessment: Examples of practice (FAST) (A work product initiated and led by Caroline Wylie, ETS, for the Formative Assessment for Students and Teachers [FAST] Collaborative). Washington, DC: Author.
5.
DowningS.HaladynaT. M. (2009). Validity and its threats. In DowningS. M.YudhowskyR. (Eds.), Assessment in the health professions (pp. 21-55). New York, NY: Routledge.
6.
FrühwirthT.AbdennadherS. (2003). Essentials of constraint programming. New York, NY: Springer.
7.
GierlM. J.HaladynaT. (2013). Automatic item generation: Theory and practice. New York, NY: Routledge.
8.
GierlM. J.LaiH. (2013). Using automated processes to generate test items. Educational Measurement: Issues and Practice, 32, 36-50.
9.
GierlM. J.LaiH. (2016a). Automatic item generation. In LaneS.RaymondM.HaladynaT. (Eds.), Handbook of test development (2nd ed., pp. 410-429). New York, NY: Routledge.
10.
GierlM. J.LaiH. (2016b). The role of cognitive models in automatic item generation. In RuppA.LeightonJ. (Eds.), Handbook of cognition and assessment: Frameworks, methodologies, and applications (pp. 124-145). New York, NY: John Wiley.
11.
GierlM. J.LaiH. (2016). A process for reviewing and evaluating generated test items. Educational Measurement: Issues and Practice, 35, 6-20. doi:10.1111/emip.12129
12.
GierlM. J.LaiH.HoganJ.MatovinovicD. (2015). A method for generating test items that are aligned to the Common Core State Standards. Journal of Applied Testing Technology, 16, 1-18.
13.
GierlM. J.LaiH.TurnerS. (2012). Using automatic item generation to create multiple-choice items for assessments in medical education. Medical Education, 46, 757-765.
14.
GierlM. J.MacMahon-BallM.VeleV.LaiH. (2015). Method for generating nonverbal reasoning items using n-layer modeling. In RasE.Joosten-ten BrinkeD. (Eds.), Proceedings from the 2015 International Computer Assisted Assessment Conference, Communications in Computer and Information Science
(pp. 1-10). New York, NY: Springer. doi:10.1007/978-3-319-27704-2_2
15.
GierlM. J.ZhouJ.AlvesC. (2008). Developing a taxonomy of item model types to promote assessment engineering. Journal of Technology, Learning, and Assessment, 72(2), 1-52.
16.
HaladynaT. M.RodriguezM. C. (2013). Developing and validating test items. New York, NY: Routledge.
17.
HattieJ.TimperleyH. (2007). The power of feedback. Review of Educational Research, 77, 81-112.
18.
HwangG. J.ChangH. F. (2011). A formative assessment-based mobile learning approach to improving the learning attitudes and achievements of students. Computers & Education, 56, 1023-1031.
19.
IrvineS. H.KyllonenP. C. (2002). Item generation for test development
. Hillsdale, NJ: Lawrence Erlbaum.
20.
KlugerA. N.DeNisiA. (1996). The effects of feedback interventions on performance: A historical review, a meta-analysis, and preliminary feedback intervention theory. Psychological Bulletin, 119, 254-284.
21.
LaDucaA.StaplesW. I.TempletonB.HolzmanG. B. (1986). Item modeling procedures for constructing content-equivalent multiple-choice questions. Medical Education, 20, 53-56.
22.
LaiH.GierlM. J.ByrneB. E.SpielmanA.WaldschmidtD. (2016). Three modeling applications to promote automatic item generation for examinations in dentistry. Journal of Dental Education, 80, 339-347.
23.
LaiH.GierlM. J.TouchieC.PughD.BoulaisA.De ChamplainA. (2016). Using automatic item generation to improve the quality of MCQ distractors. Teaching and Learning in Medicine, 28, 166-173.
24.
LaneS.RaymondM.HaladynaR. (2016). Test development process. In LaneS.RaymondM.HaladynaT. (Eds.), Handbook of test development (2nd ed., pp. 3-18). New York, NY: Routledge.
25.
PageG.BordageG. (1995). The Medical Council of Canada’s key features project: A more valid written examination of clinical decision-making skills. Academic Medicine, 70, 104-110.
26.
PerieM.HuffK. (2016). Determining content and cognitive demands for achievement tests. In LaneS.RaymondM.HaladynaT. (Eds.), Handbook of test development (2nd ed., pp. 119-143). New York, NY: Routledge.
27.
PophamJ. (2011). Transformative assessment in action: An inside look at applying the process. Alexandria, VA: Association for Supervision & Curriculum Development.
28.
RasE.WhitelockD.KalzM. (2016). The promise and potential of e-assessment for learning. In ReimannP.BullS.Kickmeier-RustM.VatrapuR.WassonB. (Eds.), Measuring and visualising learning in the information-rich classroom (pp. 21-40). London, England: Routledge.
29.
SadlerR. (1989). Formative assessment and the design of instructional systems. Instructional Science, 18, 119-144.
30.
SchmeiserC. B.WelchC. J. (2006). Test development. In BrennanR. L. (Ed.), Educational measurement (4th ed., pp. 307-353). Westport, CT: National Council on Measurement in Education and American Council on Education.
31.
ShuteV. J. (2008). Focus on formative feedback. Review of Educational Research, 78, 153-189.
