One of the most challenging aspects of writing multiple-choice test questions is identifying plausible incorrect response options—i.e., distractors. To help with this task, a procedure is introduced that can mine existing item banks for potential distractors by considering the similarities between a new item’s stem and answer and the stems and response options for items in the bank. This approach uses natural language processing to measure similarity and requires a substantial pool of items for constructing the generating model. The procedure is demonstrated with data from the United States Medical Licensing Examination (USMLE®). For about half the items in the study, at least one of the top three system-produced candidates matched a human-produced distractor exactly; and for about one quarter of the items, two of the top three candidates matched human-produced distractors. A study was conducted in which a sample of system-produced candidates were shown to 10 experienced item writers. Overall, participants thought about 81% of the candidates were on topic and 56% would help human item writers with the task of writing distractors.
BeelJ.GippB.LangerS.BreitingerC. (2016). Paper recommender systems: A literature survey. International Journal on Digital Libraries, 17(4), 305–338.
4.
BejarI. I.LawlessR.MorleyM. E.WagnerM. E.BennettR. E.RevueltaJ. (2003). A feasibility study of on-the-fly item generation in adaptive testing. Journal of Technology, Learning, and Assessment, 2(3) 1–29.
de la TorreJ. (2009). A cognitive diagnosis model for cognitively based multiple-choice options. Applied Psychological Measurement, 33, 163–183. https://doi.org/10.1177/0146621608320523
8.
FremerJ.AnastasioE. J. (1969). Computer assisted item writing (spelling items). Journal of Educational Measurement, 6, 69–74.
9.
GierlM. J.BulutO.GuoQ.ZhangX. (2017). Developing, analyzing, and using distractors for multiple-choice tests in education: A comprehensive review. Review of Educational Research, 87(6), 1082–1116.
10.
GierlM. J.LaiH. (2013a). Evaluating the quality of medical multiple-choice items created with automated processes. Medical Education, 47, 726–733.
11.
GierlM. J.LaiH. (2013b). Using automated processes to generate test items. Educational Measurement: Issues and Practice, 32, 36–50.
12.
GierlM. J.LaiH.HoganJ. B.MatovinovicD. (2015). A method for generating educational test items that are aligned to the common core state standards. Journal of Applied Testing Technology, 16(1), 1–18.
13.
GierlM. J.LaiH.PughD.TouchieC.BoulaisA-P.De ChamplainA. (2016). Evaluating the psychometric characteristics of generated multiple-choice test items. Applied Measurement in Education, 29, 196–210.
14.
HaL. A.YanevaV. (2018, June). Automatic distractor suggestion for multiple-choice tests using concept embeddings and information retrieval. In Proceedings of the thirteenth workshop on innovative use of NLP for building educational applications (pp. 389–398). https://doi.org/10.1145/3192714.3192819
15.
HaladynaT. M.RodriguezM. C.StevensC. (2019). Are multiple-choice items too fat?Applied Measurement in Education, 32(4), 350–364.
16.
HoffmanJ. I. (2015). Biostatistics for medical and biomedical practitioners. Academic press.
17.
JurafskyD.MartinJ. H. (2009). Speech and language processing: An introduction to natural language processing, computational linguistics, and speech recognition (2nd ed.). Prentice-Hall.
18.
LaiH.GierlM. J.TouchieC.PughD.BoulaisA-P.De ChamplainA. (2016). Using automatic item generation to improve the quality of MCQ distractors. Teaching and Learning in Medicine, 28, 166–173.
19.
LeoJ.KurdiG.MatentzogluN.ParsiaB.SattlerU.ForgeS.DonatoG.DowlingW. (2019). Ontology-based generation of medical, multi-term MCQs. International Journal of Artificial Intelligence in Education, 29(2), 145–188https://doi.org/10.1007/s40593-018-00172-w
MitkovR.HaL. A. (2003). Computer-aided generation of multiple-choice tests. In Proceedings of the HLT-NAACL 03 workshop on building educational applications using natural language processing (Vol. 2, pp. 17–22). Association for Computational Linguistics. https://aclanthology.org/W03-0203.pdf
22.
MitkovR.HaL. A.KaramanisN. (2006). A computer-aided environment for generating multiple-choice test items. Natural Language Engineering, 12, 177–194.
23.
PughD.De ChamplainA.GierlM.LaiH.TouchieC. (2020). Can automated item generation be used to develop high quality MCQs that assess application of knowledge?Research and Practice in Technology Enhanced Learning, 15, 12. https://doi.org/10.1186/s41039-020-00134-8
24.
RichardsJ. M.Jr (1967). Can computers write college admissions tests?Journal of Applied Psychology, 51, 211–215.
von DavierM. (2018). Automated item generation with recurrent neural networks. Psychometrika, 83, 847–857.
27.
von DavierM. (2019). Training Optimus Prime, M.D.: Generating medical certification items by fine-tuning OpenAI’s gpt2 transformer model. arXiv:1908.08594.
