Abstract
Aim
To evaluate and compare the performance of five artificial intelligence (AI) chatbots—ChatGPT (OpenAI 4), Google Gemini, Grok (xAI), DeepSeek, and Meta Llama —in delivering accurate, clear, educational, and safe responses to caregiver-facing queries related to strabismus.
Methods
Sixteen standardized caregiver questions on strabismus were presented to each chatbot in independent sessions. Five fellowship-trained pediatric ophthalmologists rated each response across four domains—Accuracy, Clarity, Educational Value, and Safety—using a 5-point Likert scale (1 = Poor, 5 = Excellent). Between-chatbot differences were analyzed using cumulative link mixed models (CLMMs) with odds ratios (OR) and 95% confidence intervals (CI). Holm-adjusted pairwise contrasts corrected for multiple comparisons. Inter-rater reliability was assessed using quadratic-weighted Fleiss’ κ and Gwet's AC1 to address prevalence and bias effects.
Results
ChatGPT achieved the highest proportion of top ratings (≥4) for Accuracy (65%) and Clarity (59%), followed by Llama (41% and 47.5%, respectively). For Educational Value, Llama (43.8%) and Gemini (42.5%) performed slightly better, while Safety ratings were highest for Gemini (40%) and Llama (37.5%). CLMM analysis showed significant between-chatbot differences for Accuracy, Clarity, and Educational Value (p < 0.05) but not for Safety. Compared with ChatGPT, lower odds of higher ratings were seen for Grok (OR 0.48) and DeepSeek (OR 0.61). Inter-rater reliability indicated moderate agreement (Fleiss’ κ = 0.59) and strong consensus (Gwet's AC1 = 0.87).
Conclusion
ChatGPT showed superior accuracy and clarity, while Gemini and Llama excelled in educational value and safety. High expert agreement supports AI chatbots as adjuncts in pediatric ophthalmology education requiring continued validation.
Keywords
Get full access to this article
View all access options for this article.
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
