Abstract
Background/Objectives
Patient information can influence decision-making and engagement with healthcare. This study compares the quality of cataract surgery patient information leaflets (PILs) generated by ChatGPT (an AI model) and two reputable hospitals, assessing AI's potential in producing high-quality patient information.
Subjects/Methods
15 ophthalmologists and 32 patients evaluated three anonymised cataract PILs: one generated by ChatGPT, one from Mount Sinai Hospital (USA) and Manchester Royal Eye Hospital (UK). Doctors used the DISCERN tool (16 questions) for quality assessment. Patients used a shortened version (5 questions). Additional preference and readability questions were added, alongside a readability assessment. PIL ratings and differences between doctor and patient scores were compared.
Results
The ChatGPT PIL scored lowest amongst doctors (mean 42.75 (SD 9.06)/75), followed by Manchester (47.04 (8.56)/75), with Mount Sinai's PIL highest (54.65 (7.09)/75) (p=<0.01). Patients similarly rated ChatGPT lowest (mean total score 4.50 (0.21)/5), with Manchester highest (4.84 (0.06)/5) (p = 0.04). Despite this, doctors were evenly divided on their preferred PIL, while more patients preferred ChatGPT over Mount Sinai. Mount Sinai's PIL had the highest inter-rater reliability(k = 0.38, 95% CI 0.10–0.60), and ChatGPT the lowest (k = 0.13, 95% CI 0.10–0.15). ChatGPT had the lowest Flesch Reading Ease score but doctors rated it most readable.
Conclusions
This study is the first to assess AI-generated cataract PILs using doctor and patient feedback. While ChatGPT received the lowest ratings, some favoured it, particularly for its clarity and readability. Doctors’ highest-rated PIL was the patients’ least favoured. This study highlights AIs potential in PIL development and the importance of doctor and patient feedback in this process.
Keywords
Get full access to this article
View all access options for this article.
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
