Evaluating Artificial Intelligence-Generated Multiple-Choice Questions in Wilderness Medicine: Quality,Feasibility,and Time Savings for Educators

Abstract

Introduction

Wilderness medicine (WM) multiple-choice questions (MCQs) are a tool used for education and assessment. This study evaluates MCQs generated by a general large language model artificial intelligence (AI), a grounded AI (based on WM resources), and human experts while also assessing potential time savings for educators.

Methods

A randomized 45-question test was created from a general Gemini AI, a grounded Gemini AI, and human written questions (with 15 questions from each source). The grounded Gemini AI was a version grounded on Auerbach's Wilderness Medicine and Wilderness Medical Society guidelines. The human written questions were obtained from prior Medical Wilderness Adventure Races because these questions were written and peer reviewed by medical educators. All questions pertained to 5 core WM topics: hypothermia, drowning, heat illness, altitude medicine, and dive/marine medicine. This test was distributed to members of WM committees with a focus on education to both take the test and rate the questions. Correct responses and ratings for accuracy, clarity, complexity, and relevance were recorded on a 5-point Likert scale. Generalized estimating equations and the 2 one-sided tests framework were used for analysis.

Results

Most respondents were attending-level emergency medicine physicians with at least 5 y of WM experience. Respondents were significantly more likely to correctly answer questions written by both general AI and grounded AI compared with human-written questions. General AI and human-written questions differed significantly with regard to accuracy, clarity, and complexity but not relevance. No statistically significant differences were found between grounded AI and human-written questions on any of the 5 metrics.

Conclusions

AI-generated WM MCQs do not significantly differ from human-written questions across all metrics, including relevance, accuracy, clarity, and complexity. Using AI-generated MCQs edited by a human expert may offer educators high-caliber MCQs with potential time savings, provided that the difficulty level of the questions is sufficiently high.