Multiple-choice item (MCI) assessments are burdensome for instructors to develop. Artificial intelligence (AI, e.g., ChatGPT) can streamline the process without sacrificing quality. The quality of AI-generated MCIs and human experts is comparable. However, whether the quality of AI-generated MCIs is equally good across various domain- and task-specific prompts remains to be determined. Therefore, we ask whether AI can generate high-quality MCIs to assess learning outcomes from a psychology textbook chapter reading.
Objective
In an exploratory study, we enlist Item Response Theory analysis and expert reviewers to assess MCIs generated by ChatGPT-4 from a psychology textbook chapter.
Method
We submitted a prompt and textbook chapter to ChatGPT-4 requesting 20 MCIs. One hundred ninety undergraduate participants read the chapter before responding to the MCIs. Expert reviewers assessed the MCIs for learning outcome alignment and quality.
Results
ChatGPT-4-generated MCIs were low in difficulty and high in discrimination. Expert reviewers found that nearly all items were logically sound, aligned with learning objectives, and met prevailing standards of MCI quality.
Conclusion
When carefully prompted, ChatGPT-4 can rapidly generate high-quality MCIs to test comprehension of a psychology textbook chapter. However, due to the uniformly low difficulty of the items, we recommend enlisting ChatGPT-4 to write MCIs for formative, but not summative, assessments.
AttaliY.Bar-HillelM. (2003). Guess where: The position of correct answers in multiple-choice test items as a psychometric variable. Journal of Educational Measurement,40(2), 109–128. https://doi.org/10.1111/j.1745-3984.2003.tb01099.x
2.
BhandariS.LiuY.PardosZ. A. (2023). Evaluating ChatGPT-generated textbook questions using IRT [Paper presentation]. Generative AI for Education Workshop (GAIED) at the Thirty-seventh Conference on Neural Information Processing Systems (NeurIPS), New Orleans, LA, December 10–16, 2023 (pp. 1–9). https://gaied.org/neurips2023/files/44/44_paper.pdf
3.
BrannanD.MohrC. D. (2024). Love, friendship, and social support. In Biswas-DienerR.DienerE. (Eds.), Noba textbook series: Psychology. DEF Publishers. http://noba.to/s54tmp7k
4.
CelikI.DindarM.MuukkonenH.JärveläS. (2022). The promises and challenges of artificial intelligence for teachers: A systematic review of research. TechTrends,66(4), 616–630. https://doi.org/10.1007/s11528-022-00715-y
5.
ChenW.-H.ThissenD. (1997). Local dependence indexes for item pairs using item response theory. Journal of Educational and Behavioral Statistics,22(3), 265–289. https://doi.org/10.3102/10769986022003265
6.
CheungB. H. H.LauG. K. K.WongG. T. C.LeeE. Y. P.KulkarniD.SeowC. S.WongR.CoM. T. H. (2023). ChatGPT versus human in generating medical graduate exam multiple-choice questions: A multinational prospective study (Hong Kong SAR, Singapore, Ireland, and the United Kingdom). PLoS One,18(8), Article e0290691. https://doi.org/10.1371/journal.pone.0290691
7.
DowningS. M. (2006a). Selected-response item formats in test development. In DowningS. M.HaladynaT. M. (Eds.), Handbook of test development (pp. 287–301). Taylor & Francis. https://doi.org/10.4324/9780203874776.ch12
8.
DowningS. M. (2006b). Twelve steps for effective test development. In DowningS. M.HaladynaT. M. (Eds.), Handbook of test development (pp. 3–25). Taylor & Francis.
9.
EbelR. L.FrisbieD. A. (1972). Evaluating tests and item characteristics. In EbelR. L.FrisbieD. A. (Eds.), Essentials of educational measurement (pp. 220–231). Prentice Hall.
HaladynaT. M.DowningS. M.RodriguezM. C. (2002). A review of multiple-choice item-writing guidelines for classroom assessment. Applied Measurement in Education,15(3), 309–333. https://doi.org/10.1207/S15324818AME1503_5
12.
HattleJ. (1985). Methodology review: Assessing unidimensionality of test and items. Applied Psychological Measurement, 9(2), 139–164. https://doi.org/10.1177/014662168500900204.
13.
HsuY. F.TsaiW. Y. (2007). EZLID: SAS macro for exact lower confidence intervals for indirect effects [Computer software]. https://www.lexjansen.com/nesug/nesug07/sa/sa10.pdf
KıyakY. S. (2023). A ChatGPT prompt for writing case-based multiple-choice questions. Revista Española de Educación Médica,4(3), 98–103. https://doi.org/10.6018/edumed.587451
16.
KrienertJ. L.WalshJ. A.CannonK. D. (2022). Changes in the tradecraft of cheating: Technological advances in academic dishonesty. College Teaching, 70(3), 309–318. https://doi.org/10.1080/87567555.2021.1940813
17.
KurdiG.LeoJ.ParsiaB.SattlerU.Al-EmariS. (2020). A systematic review of automatic question generation for educational purposes. International Journal of Artificial Intelligence in Education,30(1), 121–204. https://doi.org/10.1007/s40593-019-00186-y
18.
LordF. M. (1968). An analysis of the verbal scholastic aptitude test using Birnbaum’s three-parameter logistic model. Educational and Psychological Measurement,28(4), 989–1020. https://doi.org/10.1177/001316446802800401
19.
NwadinigweP. I.NaibiL. (2013). The number of options in a multiple-choice test item and the psychometric characteristics. Journal of Education and Practice,4(27), 189–196.
ParmenterD. A. (2009). Essay versus multiple-choice: Student preferences and the underlying rationale with implications for test construction. Academy of Educational Leadership Journal, 13(2), 57–71. https://api.semanticscholar.org/CorpusID:142548007
ReckaseM. D. (1979). Unifactor latent trait models applied to multifactor tests: Results and implications. Journal of Educational Statistics, 4(3), 207–230. https://doi.org/10.3102/10769986004003207
25.
SahinA.AnilD. (2017). The effects of test length and sample size on item parameters in item response theory. Educational Sciences: Theory & Practice,17(1), 321–335. https://doi.org/10.12738/estp.2017.1.0270
26.
SAS Institute Inc. (2024). SAS/STAT (Version 15.3) [Computer software]. https://www.sas.com
27.
SireciS. G.WileyA.KellerL. A. (1998, October 18–20). An empirical evaluation of selected multiple-choice item writing guidelines [Paper presentation]. Annual Meeting of the Northeastern Educational Research Association, Ellenville, NY. https://eric.ed.gov/?id=ED428122
28.
WhisenhuntB. L.CatheyC. L.HudsonD. L.NeedyL. M. (2022). Maximizing learning while minimizing cheating: New evidence and advice for online multiple-choice exams. Scholarship of Teaching and Learning in Psychology,8(2), 140–153. https://doi.org/10.1037/stl0000242
29.
XuX.KauerS.TupyS. (2016). Multiple-choice questions: Tips for optimizing assessment in-seat and online. Scholarship of Teaching and Learning in Psychology,2(2), 147–158. https://doi.org/10.1037/stl0000062
Yi-HsinC.LiI. (2015). IA_CTT: A SAS Macro for conducting item analysis based on classical test theory. Southeast SAS Users Group 2015 Proceedings. SAS Institute. https://www.lexjansen.com/sesug/2015/184_Final_PDF.pdf
32.
YoungR.CourtneyE.KahA.WilkersonM.Yi-HsinC. (2024). Supplementary materials for Content and item response theory analysis of ChatGPT-4-generated multiple-choice items [Supplemental Materials]. Open Science Framework. https://doi.org/10.17605/OSF.IO/ZQ4EG
