Abstract
Introduction/Purpose:
Many patients use the internet as a source for medical information regarding their medical condition. However, the accuracy and reliability of the information is variable. The recent release of general population facing artificial intelligence (AI) chatbot programs has created a potential alternative medical information source for patients. Our study examined the accuracy of the information provided by ChatGPT for total ankle arthroplasty (TAA), as determined by four fellowship-trained foot and ankle orthopedic surgeons.
Methods:
GPT-4, the latest ChatGPT model, was asked the 12 of the most common questions patients have regarding TAA. Its responses were recorded and evaluated by four fellowship-trained foot and ankle orthopedic surgeons on a scale of 1 – 4, with 1 representing an excellent response not requiring clarification and 4 representing an unsatisfactory response requiring significant clarification. Averages of scores for each question were recorded and an average of every score was used to develop an overall accuracy score.
Results:
The overall accuracy score of GPT-4 was 1.35. The responses by GPT-4 received a score of 1 for 32 of 48 (66.7%) responses. There was found to be no significant difference in scores for any of GPT-4’s responses.
Conclusion:
Our study found that GPT-4 performed well in providing accurate and near complete information to commonly questions asked to physicians by patients regarding TAA. Our results suggest that AI models, like GPT-4, may be an effective alternative source of medical information for patients. Further study with subsequent AI models and direct patient interaction may shed further light on the utility of this potential patient education modality.
ChatGPT Response Accuracy Score
Accuracy scores for ChatGPT, as determined by four foot and ankle fellowship-trained orthopedic surgeons