Competencies of Large Language Models About Piriformis Syndrome: Quality,Accuracy,Completeness,and Readability Study

Abstract

Background:The proliferation of artificial intelligence has led to widespread patient use of large language models (LLMs). Purpose: We sought to characterize LLM responses to questions about piriformis syndrome (PS). Methods: On August 15, 2024, we asked 3 LLMs—ChatGPT-4, Copilot, and Gemini—to respond to the 25 most frequently asked questions about PS, as tracked by Google Trends. We evaluated the accuracy and completeness of the responses according to the Likert scale. We used the Ensuring Quality Information for Patients (EQIP) tool to assess the quality of the responses and assessed readability using Flesch–Kincaid Reading Ease (FKRE) and Flesch–Kincaid Grade Level (FKGL) scores. Results: The mean completeness scores of the responses obtained from ChatGPT, Copilot, and Gemini were 2.8 ± 0.3, 2.2 ± 0.6, and 2.6 ± 0.4, respectively. There was a significant difference in the mean completeness score among LLMs. In pairwise comparisons, ChatGPT and Gemini were superior to Copilot. There was no significant difference between the LLMs in terms of mean accuracy scores. In readability analyses, no significant difference was found in terms of FKRE scores. However, a significant difference was found in FKGL scores. A significant difference between LLMs was identified in the quality analysis performed according to EQIP scores. Conclusion: Although the use of LLMs in healthcare is promising, our findings suggest that these technologies need to be improved to perform better in terms of accuracy, completeness, quality, and readability on PS for a general audience.