Despite National Institutes of Health and American Medical Association recommendations to publish online patient education materials at or below sixth-grade literacy, those pertaining to endoscopic skull base surgery (ESBS) have lacked readability and quality. ChatGPT is an artificial intelligence (AI) system capable of synthesizing vast internet data to generate responses to user queries but its utility in improving patient education materials has not been explored.
Objective
To examine the current state of readability and quality of online patient education materials and determined the utility of ChatGPT for improving articles and generating patient education materials.
Methods
An article search was performed utilizing 10 different search terms related to ESBS. The ten least readable existing patient-facing articles were modified with ChatGPT and iterative queries were used to generate an article de novo. The Flesch Reading Ease (FRE) and related metrics measured overall readability and content literacy level, while DISCERN assessed article reliability and quality.
Results
Sixty-six articles were located. ChatGPT improved FRE readability of the 10 least readable online articles (19.7 ± 4.4 vs. 56.9 ± 5.9, p < 0.001), from university to 10th grade level. The generated article was more readable than 48.5% of articles (38.9 vs. 39.4 ± 12.4) and higher quality than 94% (51.0 vs. 37.6 ± 6.1). 56.7% of the online articles had “poor” quality.
Conclusions
ChatGPT improves the readability of articles, though most still remain above the recommended literacy level for patient education materials. With iterative queries, ChatGPT can generate more reliable and higher quality patient education materials compared to most existing online articles and can be tailored to match readability of average online articles.
HalladayJRDonahueKECenéCW, et al.The association of health literacy and blood pressure reduction in a cohort of patients with hypertension: the heart healthy Lenoir trial. Patient Educ Couns. 2017;100(3):542-549. doi:10.1016/j.pec.2016.10.015
2.
SchillingerDGrumbachKPietteJ, et al.Association of health literacy with diabetes outcomes. JAMA. 2002;288(4):475-482. doi:10.1001/jama.288.4.475
3.
BerkmanNDSheridanSLDonahueKEHalpernDJCrottyK. Low health literacy and health outcomes: an updated systematic review. Ann Intern Med. 2011;155(2):97-107. doi:10.7326/0003-4819-155-2-201107190-00005
4.
KutnerMGreenburgEJinYPaulsenC. 2006. The Health Literacy of America’s Adults Results From the 2003 National Assessment of Adult Literacy. ED Pubs, Jessup, MD. Web site: http://www.edpubs.org.
5.
MurrayMTuWWuJMorrowDSmithFBraterD. Factors associated with exacerbation of heart failure include treatment adherence and health literacy skills. Clin Pharmacol Ther. 2009;85(6):651-658. doi:10.1038/clpt.2009.7
6.
TassonePGeorgalasCPatelNNApplebyEKotechaB. Do otolaryngology out-patients use the internet prior to attending their appointment?J Laryngol Otol.2004;118(1):34-38. doi:10.1258/002221504322731600
7.
TeplinskyEPonceSBDrakeEK, et al.Online medical misinformation in cancer: distinguishing fact from fiction. JCO Oncol Pract. 2022;18(8):584-589. doi:10.1200/op.21.00764
8.
Swire-ThompsonBLazerD. Public health and online misinformation: challenges and recommendations. Annu Rev Public Health. 2019;41:433-451. doi:10.1146/annurev-publhealth-040119-094127
9.
YeungANgEAbi-JaoudeE. TikTok and attention-deficit/hyperactivity disorder: a cross-sectional study of social media content quality. Can J Psychiatry.2022;67(12):899-906. doi:10.1177/07067437221082854
10.
RobertsBKobritzMNofiC, et al.Social media, misinformation, and online patient education in emergency general surgical procedures. J Surg Res.2023;287:16-23. doi:10.1016/j.jss.2023.01.009
11.
BitnerBFGowdaSMarkMEWarnerDCTajudeenBAKuanEC. See many, do one, teach many more: assessing quality and reliability of publicly available endoscopic videos in rhinology. Int Forum Allergy Rhinol. 2022;12(12):1527-1534. doi:10.1002/alr.23006
12.
FersterAPOCHuA. Evaluating the quality and readability of internet information sources regarding the treatment of swallowing disorders. Ear Nose Throat J. 2017;96(3):128-138. doi:10.1177/014556131709600312
13.
GroseEMHolmesCPAravinthanKAWuVLeeJM. Readability and quality assessment of internet-based patient education materials related to nasal septoplasty. J Otolaryngol Head Neck Surg. 2021;50(1). doi:10.1186/S40463-021-00507-Z
14.
MisraPAgarwalNKasabwalaKHansberryDRSetzenMEloyJA. Readability analysis of healthcare-oriented education resources from the American Academy of Facial Plastic and Reconstructive Surgery. Laryngoscope. 2013;123(1):90-96. doi:10.1002/lary.23574
15.
JoyPSantosFDaarDAPaydarKZWirthGA. Readability of online materials for rhinoplasty. World J Plast Surg. 2018;7(1):89. /pmc/articles/PMC5890371/.
16.
KongKAHuA. Readability assessment of online tracheostomy care resources. Otolaryngol Head Neck Surg. 2015;152(2):272-278. doi:10.1177/0194599814560338
17.
BalakrishnanVChandyZHseihABuiTLVermaSP. Readability and understandability of online vocal cord paralysis materials. Otolaryngol Head Neck Surg (United States). 2016;154(3):460-464. doi:10.1177/0194599815626146
18.
KasabwalaKMisraPHansberryDR, et al.Readability assessment of the American Rhinologic Society patient education materials. Int Forum Allergy Rhinol. 2013;3(4):325-333. doi:10.1002/alr.21097
19.
TanJYTanYCYapD. Readability and quality of online patient health information on parotidectomy. J Laryngol Otol.2023;137(12):1378-1383. doi:10.1017/S0022215123000336
20.
San GiorgiMRMde GrootOSDDikkersFG. Quality and readability assessment of websites related to recurrent respiratory papillomatosis. Laryngoscope. 2017;127(10):2293-2297. doi:10.1002/lary.26521
21.
TingKHuA. Evaluating the quality and readability of thyroplasty information on the internet. J Voice.2014;28(3):378-381. doi:10.1016/j.jvoice.2013.10.011
22.
PatelCRSanghviSCherlaDVBaredesSEloyJA. Readability assessment of internet-based patient education materials related to parathyroid surgery. Ann Otol Rhinol Laryngol. 2015;124(7):523-527. doi:10.1177/0003489414567938
23.
ChiEJabbourNAaronsonNL. Quality and readability of websites for patient information on tonsillectomy and sleep apnea. Int J Pediatr Otorhinolaryngol. 2017;98:1-3. doi:10.1016/j.ijporl.2017.04.031
24.
YiGSHuA. Quality and readability of online information on in-office vocal fold injections. Ann Otol Rhinol Laryngol. 2020;129(3):294-300. doi:10.1177/0003489419887406
25.
CherlaDVSanghviSChoudhryOJLiuJKEloyJA. Readability assessment of internet-based patient education materials related to endoscopic sinus surgery. Laryngoscope. 2012;122(8):1649-1654. doi:10.1002/lary.23309
26.
SaxLRazakAShettyKCohenMLeviJ. Readability of online patient education materials for parents after a failed newborn hearing screen. Int J Pediatr Otorhinolaryngol. 2019;125:168-174. doi:10.1016/j.ijporl.2019.07.009
27.
SviderPFAgarwalNChoudhryOJ, et al.Readability assessment of online patient education materials from academic otolaryngology–head and neck surgery departments. Am J Otolaryngol. 2013;34(1):31-35. doi:10.1016/J.AMJOTO.2012.08.001
28.
GroseEMHolmesCPAravinthanKAWuVLeeJM. Readability and quality assessment of internet-based patient education materials related to nasal septoplasty. J Otolaryngol Head Neck Surg. 2021;50(1). doi:10.1186/s40463-021-00507-z
29.
WeissBDSchwartzbergJGDavisTCParkerRMSokolPEWilliamsMV. Health literacy and patient safety: help patients understand: manual for clinicians.American Medical Association; 2007.
30.
DaveTAthaluriSASinghS. ChatGPT in medicine: an overview of its applications, advantages, limitations, future prospects, and ethical considerations. Front Artif Intell. 2023;6. doi:10.3389/frai.2023.1169595
31.
OpenAI. 2023. ChatGPT: Optimizing Language Models for Dialogue. Available online at: https://openai.com/blog/chatgpt/ (Accessed Sept 2, 2023)2.
KasabwalaKMisraPHansberryDR, et al.Readability assessment of the American Rhinologic Society patient education materials. Int Forum Allergy Rhinol. 2013;3(4):325-333. doi:10.1002/ALR.21097
34.
HoangPMvan BallegooieC. Assessment of the readability and quality of online patient education material for chronic medical conditions. Healthcare (Switzerland). 2022;10(2):234. doi:10.3390/healthcare10020234
35.
MisraPAgarwalNKasabwalaKHansberryDRSetzenMEloyJA. Readability analysis of healthcare-oriented education resources from the American Academy of Facial Plastic and Reconstructive Surgery. Laryngoscope. 2013;123(1):90-96. doi:10.1002/lary.23574
36.
MisraPAgarwalNKasabwalaKHansberryDRSetzenMEloyJA. Readability analysis of healthcare-oriented education resources from the American Academy of Facial Plastic and Reconstructive Surgery. Laryngoscope. 2013;123(1):90-96. doi:10.1002/LARY.23574
CharnockDShepperdSNeedhamGGannR. DISCERN: an instrument for judging the quality of written consumer health information on treatment choices. J Epidemiol Community Health (1978). 1999;53(2):105-111. doi:10.1136/JECH.53.2.105
39.
MoultonLSEvansPAStarksISmithT. Pre-operative education prior to elective hip arthroplasty surgery improves postoperative outcome. Int Orthop. 2015;39(8):1483-1486. doi:10.1007/s00264-015-2754-2
40.
FuMRChenCMHaberJGuthAAAxelrodD. The effect of providing information about lymphedema on the cognitive and symptom outcomes of breast cancer survivors. Ann Surg Oncol. 2010;17(7):1847-1853. doi:10.1245/s10434-010-0941-3
41.
BajwaJMunirUNoriAWilliamsB. Artificial intelligence in healthcare: transforming the practice of medicine. Future Healthc J. 2021;8(2):e188-e194. doi:10.7861/fhj.2021-0095
42.
ElseH. Abstracts written by ChatGPT fool scientists. Nature. 2023;613(7944):423-423. doi:10.1038/d41586-023-00056-7
43.
KitamuraFC. ChatGPT is shaping the future of medical writing but still requires human judgment. Radiology. 2023;307(2). doi:10.1148/radiol.230171
44.
ShenYHeacockLEliasJ, et al.ChatGPT and other large language models are double-edged swords. Radiology. 2023;307(2). doi:10.1148/RADIOL.230163
45.
BiswasS. ChatGPT and the future of medical writing. Radiology. 2023;307(2). doi:10.1148/RADIOL.223312
46.
KingMR. A conversation on artificial intelligence, chatbots, and plagiarism in higher education. Cell Mol Bioeng. 2023;16(1):1-2. doi:10.1007/s12195-022-00754-8
47.
ScottB. Learn about the Flesch Reading Ease Formula. readabilityformulas.com.
48.
WongKGiladACohenMBKirkeDNJalisiSM. Patient education materials assessment tool for laryngectomy health information. Head Neck. 2017;39(11):2256-2263. doi:10.1002/HED.24891
49.
KitamuraFC. ChatGPT is shaping the future of medical writing but still requires human judgment. Radiology. 2023;307(2). doi:10.1148/radiol.230171
50.
van DisEAMBollenJZuidemaWvan RooijRBocktingCL. ChatGPT: five priorities for research. Nature. 2023;614(7947):224-226. doi:10.1038/d41586-023-00288-7
51.
MeheraliSPunjaniNSMevawalaA. Health literacy interventions to improve health outcomes in low- and middle-income countries. Health Lit Res Pract. 2020;4(4):e251-e266. doi:10.3928/24748307-20201118-01
52.
BojrabDIFritzCBabuSLinKF. A critical analysis of the information available online for Ménière’s disease. Otolaryngol Head Neck Surg (United States). 2020;162(3):329-336. doi:10.1177/0194599819901150
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
