Medical students’ perceptions of AI-generated practice questions as learning tools

Abstract

Generative artificial intelligence (AI) tools, including large language models such as ChatGPT, have potential as educational adjuncts to enhance student learning. This study evaluated the perceived utility of and performance outcomes associated with formative, AI-generated, United States Medical Licensing Examination-style practice questions among preclinical medical students. Multiple-choice questions (MCQs) aligned with 15 microbiology and endocrinology lectures were generated with ChatGPT 4.0 and distributed via Google Forms to 386 students (198 first-year medical student (MS1), 188 second-year medical student (MS2)) at a U.S. medical school. Each question set consisted of six questions on average, and these groupings were considered individual “question sets” in our analysis. Question completion was optional for students and a total of 490 question sets were completed. Students provided feedback on 94.9% of sets, with 82.8% rating the questions as “Helpful,” 16.1% as “Somewhat Helpful,” and 1.1% as “Not Helpful.” MS2s answered a significantly higher number of questions correctly relative to MS1s (84.4% vs 78.6%, p < 0.001), and performance varied by lecture topic. However, average scores did not differ significantly based on whether students provided feedback or how they rated the questions (p = 0.587). These findings suggest that AI-generated MCQs are broadly accepted by students as supplemental formative learning tools. While performance variation was observed, conclusions about broader feasibility should be considered exploratory and warrant further study.