Artificial Intelligence in Surgical Education: A Pilot Study Using ASCRS Guideline-Derived Questions

Abstract

Background

Large language models (LLMs) have demonstrated strong performance on general medical and surgical examinations; however, their capacity to accurately interpret and apply subspecialty clinical practice guidelines remains incompletely characterized.

Objective

To evaluate and compare the accuracy and consistency of two contemporary LLMs—Google Gemini and OpenEvidence—using multiple-choice questions (MCQs) derived directly from the 2022 American Society of Colon and Rectal Surgeons (ASCRS) Clinical Practice Guidelines for anorectal abscess, fistula-in-ano, and rectovaginal fistula.

Methods

Thirty guideline-based MCQs were developed and independently validated by surgeon reviewers. Each question was presented to both models under identical conditions without additional prompting. Accuracy was calculated with 95% confidence intervals and compared against chance performance (p₀ = .25). Inter-model agreement was assessed using Cohen’s kappa coefficient.

Results

Both Gemini and OpenEvidence correctly answered 29 of 30 questions (96.7%; 95% CI, 0.83-0.999), significantly exceeding chance performance (P < .0001 for both models). Both models missed the same question, yielding perfect inter-model agreement (κ = 1.0).

Conclusion

In this focused pilot study restricted to ASCRS anorectal disease guidelines, both LLMs demonstrated near-perfect and statistically equivalent accuracy. These findings suggest that contemporary LLMs can accurately apply subspecialty surgical guidelines within a narrow domain, though broader, multi-guideline evaluations are required before generalization.