Abstract
Background
ChatGPT, the most widely recognized AI chatbot, has been increasingly utilized by patients for health care education, but its performance and knowledge base have not been evaluated for colorectal surgery topics. We hypothesize ChatGPT can provide accurate patient-level information for benign anorectal diseases.
Methods
We performed a single-institution prospective study evaluating OpenAI’s GPT-4 chatbot against verified online medical literature using the modified EQIP (mEQIP) for hemorrhoids, anal fissures, and pruritus ani. Scoring was performed by three independent educated reviewers.
Results
ChatGPT had a median overall score of 22/36 (61%) across all topics. It performed similarly for all topics: 24 for hemorrhoids, 22 for fissures, and 20 for pruritus (P = 0.15). ChatGPT was strongest within the mEQIP Content domain for all topics (P = 0.001). It performed weakest within the mEQIP Identification domain. It primarily lost points for recommending non-evidence-based treatments, lack of citations and visual aids, and generating broken source links.
Discussion
ChatGPT can provide accurate, reliable information at patient understanding level and is comparable to other validated online information for benign anorectal pathologies. It could improve on mEQIP performance by improving source documentation and visual aid capability, but it remains a promising patient resource with an intuitive user interface.
Get full access to this article
View all access options for this article.
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
