Abstract
Background
Colorectal surgery stapling misadventures are fairly common, potentially leading to serious complications. Although artificial intelligence (AI)-driven chatbots have shown promise as educational tools in various medical fields, their utility in real-time surgical decision-making remains unclear. This study assessed the ability of two chatbots, ChatGPT-4 and Google Gemini, to suggest management strategies for various stapling misadventures.
Methods
In this exploratory study, 21 stapling misadventure scenarios were developed based on a literature search and expert input. The scenarios were uploaded to both chatbots with a prompt asking about the management strategies for each scenario. ChatGPT-4 and Google Gemini’s suggestions for the scenarios were independently evaluated by 3 colorectal surgeons. The main outcome measures were appropriateness, comprehensiveness, and justification quality assessed on a 0-2 Likert scale.
Results
ChatGPT-4’s suggestions for all scenarios were rated as fully or partly appropriate and comprehensive, compared to 85.7-90.5% for Gemini’s suggestions. The median appropriateness and comprehensiveness scores were higher for ChatGPT-4 (2 vs 1), and was statistically significant according to one assessor (P = 0.005 and 0.002). The justification of ChatGPT’s suggestions was more appropriate according to 2 assessors. Assessors found that 90-95% of ChatGPT-4’s suggestions and 76-90% of Gemini’s suggestions were useful clinical aids for surgeons. ChatGPT-4’s suggestions showed better agreement among assessors on appropriateness (43% vs 24%) and justification of responses (42.8% vs 19%).
Conclusions
Both ChatGPT-4 and Google Gemini provided appropriate and comprehensive suggestions for colorectal stapling misadventures, with ChatGPT-4 showing marginally better performance. These findings support the potential role of AI-driven chatbots as decision-support tools in surgery.
Get full access to this article
View all access options for this article.
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
