Advances in Large Language Model Performance: A Comparative Study of ChatGPT-4 and ChatGPT-5 on ABSITE Questions

Abstract

Objective

This study aimed to directly compare the performance of 2 successive versions of ChatGPT (ChatGPT-4o and ChatGPT-5) in answering questions from the American Board of Surgery In-Training Examination (ABSITE) Quiz.

Methods

A total of 170 multiple-choice ABSITE Quiz questions (2017-2022) were categorized into 4 subgroups: Definitions, Biochemistry/Pharmaceutical, Case Scenario, and Treatment & Surgical Procedures. Each question was entered into both ChatGPT versions using the same question set. Correct answer rates were recorded, and paired comparisons were conducted using McNemar’s test.

Results

Overall accuracy was 79.4% for ChatGPT-4o and 87.1% for ChatGPT-5, with the improvement statistically significant (P < 0.001). In the Case Scenario category, accuracy increased from 76.3% to 86.8% (+10.5 points, P = 0.008), reflecting enhanced performance in multi-step clinical decision making. In contrast, Definitions (93.5% vs 93.5%) and Biochemistry/Pharmaceutical (83.3% vs 83.3%) showed no significant difference due to ceiling effects. In the Treatment & Surgical Procedures category, accuracy improved from 69.2% to 76.9%, but without statistical significance owing to the small sample size.

Conclusion

ChatGPT-5 demonstrated significantly higher accuracy than ChatGPT-4o in ABSITE Quiz questions, particularly in case-based scenarios requiring clinical reasoning. These findings suggest that newer LLM versions may provide more reliable support in surgical education and exam preparation, though further validation in multimodal and real exam settings is needed.