Abstract
Objectives
The purpose of this study was to use standardised scoring systems in the analysis and comparison of the quality and readability of information regarding hip fractures and their management when querying an AI software (ChatGPT).
Methods
An open AI model (ChatGPT) was used to answer 25 commonly asked questions from patients regarding hip fractures as described by three international societies. Subsequent evaluation and comparison was carried out by two reviewers for medical accuracy, quality and readability using the JAMA Benchmark criteria, DISCERN score, Flesch-Kincaid Reading Ease Score (FRES) & Grade Level (FKGL).
Results
For the ChatGPT responses, the JAMA Benchmark criteria score was 0, which is the lowest score indicating no reliable resources cited. The DISCERN score was 60, which is considered a good score. The FRES was 27.7 (a ‘very confusing’ score), and the FKGL was considered to be that of a ‘college graduate’. In contrast, the other sources analysed had a mean FRES of 70.3 (a ‘fairly easy’ score), with the FKGL considered to be that of ‘middle school student’.
Conclusion
Using ChatGPT resulted in very high quality answers given on questions relating to hip fractures and their management, although the authors noted that a much higher reading level was required to comprehend the generated information on the topic when compared to other credible sources. Thus, while the information was readily accessible, it was not easily understandable. Furthermore, the lack of citations generated is an ongoing source of concern regarding the reliability of responses. Whilst ChatGPT offers an accessible information resource for patients and their families, this study highlights the need for vetted, understandable and reliable literature to be made available by caregivers to inform our patients.
Level of Evidence
V
Get full access to this article
View all access options for this article.
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
