Evaluating ChatGPT’s Ability to Answer Common Parent Questions on Avascular Necrosis in Children

Abstract

Background:

Avascular necrosis (AVN) of the bone may result in severe pain, and patients with AVN and their families may seek out information about the condition. With the rise of ChatGPT, AVN patients and families may turn to this chatbot with questions.

Purpose:

We sought to explore expert clinicians’ perceptions of the quality of ChatGPT’s responses to frequently asked parent questions about AVN in children. Secondary aims of this study were to assess provider perceptions of ChatGPT and AVN parental education and to evaluate the readability of ChatGPT responses.

Methods:

We conducted a cross-sectional survey study of 9 pediatric orthopedic surgeons, oncologists, and advanced practice providers with expertise in the clinical management of AVN. Fifteen common questions parents ask about AVN were posed to ChatGPT, preceded by the following prompt: “Please answer the following parent question relating to avascular necrosis. Please give me a response at or below a sixth-grade reading level: [Question].” The answers were evaluated by participants using a 4-point Likert scale. ChatGPT responses were also assessed using the following readability scores: Flesch-Kincaid Grade Level, Gunning Fog index, and Flesch Reading Ease. In addition, the survey included 4 questions developed to gather overall provider perceptions.

Results:

Providers deemed answers to all 15 questions as at least satisfactory, requiring minimal clarification on average. Yet only 3 ChatGPT responses (20%) were at or below a sixth-grade reading level, as prompted. The average Flesch-Kincaid Grade Level was 6.94, and the average Gunning Fog Index was 9.22, suggesting the responses reflect a reading level between approximately seventh grade and early high school. A majority of providers agreed that these responses would be sufficient for most parents (56%) and that the information was at the appropriate reading level (100%).

Conclusion:

The findings of this small survey study suggest that ChatGPT’s responses to common parent questions about AVN were satisfactory, requiring minimal clarification. ChatGPT has the potential to serve as a resource for orthopedic patients and family education, though concerns remain.