Evaluating ChatGPT's efficacy and readability to common pediatric ophthalmology and strabismus-related questions

Abstract

Introduction

The rise in popularity of chatbots, particularly ChatGPT by OpenAI among the general public and its utility in the healthcare field is a topic of present controversy. The current study aimed at assessing the reliability and accuracy of ChatGPT's responses to inquiries posed by parents, specifically focusing on a range of pediatric ophthalmological and strabismus conditions.

Methods

Patient queries were collected via a thematic analysis and posed to ChatGPT 3.5 version across 3 unique instances each. The questions were divided into 12 domains totalling 817 unique questions. All responses were scored on the response quality by two experienced pediatric ophthalmologists in a Likert-scale format. All questions were evaluated for readability using the Flesch-Kincaid Grade Level (FKGL) and character counts.

Results

A total of 638 (78.09%) questions were scored to be perfectly correct, 156 (19.09%) were scored correct but incomplete and only 23 (2.81%) were scored to be partially incorrect. None of the responses were scored to be completely incorrect. Average FKGL score was 14.49 [95% CI 14.4004–14.5854] and the average character count was 1825.33 [95%CI 1791.95–1858.7] with p = 0.831 and 0.697 respectively. The minimum and maximum FKGL scores were 10.6 and 18.34 respectively. FKGL predicted character count, R²=.012, F(1,815) = 10.26, p = .001.

Conclusion

ChatGPT provided accurate and reliable information for a majority of the questions. The readability of the questions was much above the typically required standards for adults, which is concerning. Despite these limitations, it is evident that this technology will play a significant role in the healthcare industry.

Keywords

Pediatric ophthalmology strabismus chatbot health information artificial intelligence

Get full access to this article

View all access options for this article.

References

Bohr

Memarzadeh

. The rise of artificial intelligence in healthcare applications. In: Artificial intelligence in healthcare. Cambridge, MA: Elsevier, 2020, pp.25–60.

Hosny

Parmar

Quackenbush

, et al. Artificial intelligence in radiology. Nat Rev Cancer 2018; 18: 500–510. doi:https://doi.org/10.1038/s41568-018-0016-5

Singh

. Eyeing a role for artificial intelligence in clinical trials. JAMA Ophthalmol 2023; 141: 313. doi:https://doi.org/10.1001/jamaophthalmol.2022.6542

Dave

Athaluri

Singh

. ChatGPT in medicine: an overview of its applications, advantages, limitations, future prospects, and ethical considerations. Front Artif Intell 2023; 6: 1169595. doi:10.3389/frai.2023.1169595

Patrinos

Sarhangi

Sarrami

, et al. Using ChatGPT to predict the future of personalized medicine. Pharmacogenomics J 2023; 23: 178–184. doi:https://doi.org/10.1038/s41397-023-00316-9

Introducing ChatGPT. Openai.com. Accessed December 14, 2023. https://openai.com/blog/chatgpt/.

Abdullah

Madain

Jararweh

. ChatGPT: Fundamentals, Applications and Social Impacts. In: 2022 Ninth International Conference on Social Networks Analysis, Management and Security (SNAMS). IEEE 2022.

Frey

Bonfiglioli

Brunner

, et al. Parents’ use of social media as a health information source for their children: a scoping review. Acad Pediatr 2022; 22: 526–539. doi:https://doi.org/10.1016/j.acap.2021.12.006

Gruebner

van Haasteren

Hug

, et al. Digital platform uses for help and support seeking of parents with children affected by disabilities: scoping review. J Med Internet Res 2022; 24: e37972. doi:https://doi.org/10.2196/37972

10.

Walker

Ghani

Kuemmerli

, et al. Reliability of medical information provided by ChatGPT: assessment against clinical guidelines and patient information quality instrument. J Med Internet Res 2023; 25: e47479. doi:https://doi.org/10.2196/47479

11.

Mondal

Dash

Mondal

, et al. ChatGPT in answering queries related to lifestyle-related diseases and disorders. Cureus 2023; 15(11): e48296. doi:https://doi.org/10.7759/cureus.48296

12.

The Lancet Digital Health. ChatGPT: friend or foe? Lancet Digit Health 2023; 5: e102. doi:https://doi.org/10.1016/s2589-7500(23)00023-7

13.

Learn more about eyes. Aapos.org. Accessed December 14, 2023. https://www.aapos.org/patient/eye-terms.

14.

Datayze. Readability Analyzer. Datayze.com. Accessed December 14, 2023. https://datayze.com/readability-analyzer.

15.

Liu

, et al.

Revolutionizing neurosurgery with GPT-4: a leap forward or ethical conundrum?

Ann Biomed Eng 2023; 51: 2105–2112. doi:https://doi.org/10.1007/s10439-023-03240-y

16.

Nicholl

Tracey

Begley

, et al. Internet use by parents of children with rare conditions: findings from a study on parents’ web information needs. J Med Internet Res 2017; 19: e51. doi:https://doi.org/10.2196/jmir.5834

17.

Kubb

Foran

. Online health information seeking by parents for their children: systematic review and agenda for further research. J Med Internet Res 2020; 22: e19985. doi:https://doi.org/10.2196/19985

18.

Pehora

Gajaria

Stoute

, et al. Are parents getting it right? A survey of parents’ internet use for children’s health care information. Interact J Med Res 2015; 4: e12. doi:https://doi.org/10.2196/ijmr.3790

19.

Moon

Mathews

Oden

, et al. Mothers’ perceptions of the internet and social media as sources of parenting and health information: qualitative study. J Med Internet Res 2019; 21: e14289. doi:https://doi.org/10.2196/14289

20.

How to find reliable health information online. National Institute on Aging. Accessed December 14, 2023. https://www.nia.nih.gov/health/healthy-aging/how-find-reliable-health-information-online

21.

Johnson

Khan

Koo

. Social media and vision therapy: perspectives of providers and patients on Instagram. J AAPOS 2021; 25: 166.e1–166.e5. doi:https://doi.org/10.1016/j.jaapos.2020.11.023

22.

Huang

Abdullah

AAN

Chen

, et al. Ophthalmology and social media: an in-depth investigation of ophthalmologic content on Instagram. Clin Ophthalmol 2022; 16: 685–694. doi:https://doi.org/10.2147/opth.s353417

23.

Young

Verter

Howard

. Quality analysis of publicly available videos for pediatric strabismus surgery. J AAPOS 2020; 24: 102–104. doi:https://doi.org/10.1016/j.jaapos.2019.11.009

24.

Kaptı

Erdem

. Evaluation of the reliability and quality of YouTube videos on congenital nasolacrimal duct obstruction. Cureus 2023; 15(3): e36365. doi:https://doi.org/10.7759/cureus.36365

25.

Rau

Basir

Flynn

. Parental understanding of crucial medical jargon used in prenatal prematurity counseling. BMC Med Inform Decis Mak 2020; 20(1). doi:https://doi.org/10.1186/s12911-020-01188-w

26.

Yin

Johnson

Mendelsohn

, et al. The health literacy of parents in the United States: a nationally representative study. Pediatrics 2009; 124(Supplement_3): S289–S298. doi:https://doi.org/10.1542/peds.2009-1162E

27.

Rajah

Hassali

MAA

Murugiah

. A systematic review of the prevalence of limited health literacy in Southeast Asian countries. Public Health 2019; 167: 8–15. doi:https://doi.org/10.1016/j.puhe.2018.09.028

28.

Ikediashi

Ehrmann

Gomez

, et al. Disease knowledge and health literacy in parents of children with sickle cell disease. EJHaem 2023; 4: 956–962. doi:https://doi.org/10.1002/jha2.762

29.

Doak

Root

. Teaching patients with low literacy skills. Am J Nurs 1996; 96: 16M. doi:https://doi.org/10.1097/00000446-199612000-00022

30.

Badarudeen

Sabharwal

. Assessing readability of patient education materials: current role in orthopaedics. Clin Orthop Relat Res 2010; 468: 2572–2580. doi:https://doi.org/10.1007/s11999-010-1380-y

31.

Daraz

Morrow

Ponce

, et al. Readability of online health information: a meta-narrative systematic review. Am J Med Qual 2018; 33: 487–492. doi:https://doi.org/10.1177/1062860617751639

32.

Cheng

Fernandez

Shnorhavorian

, et al. Engagement of common pediatric urologic conditions on social media. J Pediatr Urol 2022; 18: 236.e1–236.e7. doi:https://doi.org/10.1016/j.jpurol.2021.12.003

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.07 MB

0.02 MB

0.27 MB