Background

Colorectal surgery stapling misadventures are fairly common, potentially leading to serious complications. Although artificial intelligence (AI)-driven chatbots have shown promise as educational tools in various medical fields, their utility in real-time surgical decision-making remains unclear. This study assessed the ability of two chatbots, ChatGPT-4 and Google Gemini, to suggest management strategies for various stapling misadventures.

Methods

In this exploratory study, 21 stapling misadventure scenarios were developed based on a literature search and expert input. The scenarios were uploaded to both chatbots with a prompt asking about the management strategies for each scenario. ChatGPT-4 and Google Gemini’s suggestions for the scenarios were independently evaluated by 3 colorectal surgeons. The main outcome measures were appropriateness, comprehensiveness, and justification quality assessed on a 0-2 Likert scale.

Results

ChatGPT-4’s suggestions for all scenarios were rated as fully or partly appropriate and comprehensive, compared to 85.7-90.5% for Gemini’s suggestions. The median appropriateness and comprehensiveness scores were higher for ChatGPT-4 (2 vs 1), and was statistically significant according to one assessor (P = 0.005 and 0.002). The justification of ChatGPT’s suggestions was more appropriate according to 2 assessors. Assessors found that 90-95% of ChatGPT-4’s suggestions and 76-90% of Gemini’s suggestions were useful clinical aids for surgeons. ChatGPT-4’s suggestions showed better agreement among assessors on appropriateness (43% vs 24%) and justification of responses (42.8% vs 19%).

Conclusions

Both ChatGPT-4 and Google Gemini provided appropriate and comprehensive suggestions for colorectal stapling misadventures, with ChatGPT-4 showing marginally better performance. These findings support the potential role of AI-driven chatbots as decision-support tools in surgery.

Search results

Utility of Artificial Intelligence-Driven Chatbots in Suggesting Solutions to Stapling Misadventures in Colorectal Surgery