Comparison of adhesion scoring performance between humans and large language models in experimental rat laparotomy

Abstract

This study compared the macroscopic adhesion scoring performance of large language models (LLMs: ChatGPT-o3, ChatGPT-5, Gemini-2.5 Pro) with that of novice veterinary surgeons, using expert consensus as the reference. Eighty standardized postoperative laparotomy cases in Wistar rats were photographed and scored using the Nair 0–4 adhesion scale. Two novice surgeons and three LLMs independently evaluated each case; the expert reference was defined by a surgeon and a pathologist. Group differences were analyzed using the Kruskal–Wallis test with Dunn–Bonferroni post hoc comparisons, correlations by Bonferroni-adjusted Spearman coefficients, human interobserver reliability by intraclass correlation coefficient (ICC) (A,1), and agreement with the expert by quadratic-weighted Cohen’s κ and exact-match accuracy. Overall differences were significant. ChatGPT-o3, ChatGPT-5, Gemini-2.5 Pro, and Novice 1 assigned lower scores, while Novice 2 assigned higher scores. Correlations with the expert were significant for Novice 1 (ρ = 0.706), Novice 2 (ρ = 0.593), and ChatGPT-o3 (ρ = 0.617), but not for ChatGPT-5 or Gemini-2.5 Pro. Inter-observer reliability among human raters was moderate (ICC = 0.55). Importantly, absolute exact-match accuracies were modest across all evaluators, with the highest accuracy observed for Novice 1 (33.8%) and ⩽26.3% for the LLMs. While novices outperformed the models, these findings highlight the intrinsic difficulty of fine-grained Nair 0–4 adhesion scoring on two-dimensional intraoperative images and indicate that current LLMs are better suited as calibrated decision-support tools rather than stand-alone raters.