Large Language Models Triage of Retina Patient Emergency Telephone Calls: A Pilot Study

Abstract

Purpose: To compare the diagnostic and management accuracy of large language model chatbots vs that of humans in performing outpatient retina triage in on-call telephone emergencies. Methods: Four large language model chatbots, 3 vitreoretinal surgery fellows, and 3 certified ophthalmic technicians with on-call experience were presented with 10 simulated retina cases representing after-hours telephone calls from patients. Diagnosis and triage recommendations were obtained from chatbots and humans. Recommendations were graded for each chatbot and human respondent. Results: Human graders were significantly more accurate than chatbots in diagnosis (95% vs 76.7%, respectively; P < .01) and follow-up recommendations (85% vs 70%, respectively; P = .03). However, chatbot performance varied. ChatGPT (OpenAI; 90%, P = .4) and Claude (Anthropic; 83.3%, P = .11) were noninferior to humans in diagnosis, while Meta (Meta Platforms Inc; 76.7%, P = .01) and Gemini (Google LLC; 56.7%, P < .001) performed significantly worse than humans. ChatGPT (93.3%, P = .32) and Claude (90%, P = .74) were also noninferior to humans in follow-up recommendations, but Gemini (50%, P < .001) and Meta (46.7%, P < .001) were worse than humans. Conclusions: The current pilot study found that overall, humans performed better than large language model–based chatbots in diagnosing and triaging retina-specific on-call telephone emergencies. However, chatbot accuracy was variable, with ChatGPT and Claude showing noninferior performance compared with humans. These findings suggest that with further validation, certain large language models could serve as useful aides for managing emergency telephone calls of varying medical urgency.