Feasibility and acceptability of ChatGPT generated radiology report summaries for cancer patients

Abstract

Objective

Patients now have direct access to their radiology reports, which can include complex terminology and be difficult to understand. We assessed ChatGPT's ability to generate summarized MRI reports for patients with prostate cancer and evaluated physician satisfaction with the artificial intelligence (AI)-summarized report.

Methods

We used ChatGPT to summarize five full MRI reports for patients with prostate cancer performed at a single institution from 2021 to 2022. Three summarized reports were generated for each full MRI report. Full MRI and summarized reports were assessed for readability using Flesch-Kincaid Grade Level (FK) score. Radiation oncologists were asked to evaluate the AI-summarized reports via an anonymous questionnaire. Qualitative responses were given on a 1–5 Likert-type scale. Fifty newly diagnosed prostate cancer patient MRIs performed at a single institution were additionally assessed for physician online portal response rates.

Results

Fifteen summarized reports were generated from five full MRI reports using ChatGPT. The median FK score for the full MRI reports and summarized reports was 9.6 vs. 5.0, (p < 0.05), respectively. Twelve radiation oncologists responded to our questionnaire. The mean [SD] ratings for summarized reports were factual correctness (4.0 [0.6], understanding 4.0 [0.7]), completeness (4.1 [0.5]), potential for harm (3.5 [0.9]), overall quality (3.4 [0.9]), and likelihood to send to patient (3.1 [1.1]). Current physician online portal response rates were 14/50 (28%) at our institution.

Conclusions

We demonstrate a novel application of ChatGPT to summarize MRI reports at a reading level appropriate for patients. Physicians were likely to be satisfied with the summarized reports with respect to factual correctness, ease of understanding, and completeness. Physicians were less likely to be satisfied with respect to potential for harm, overall quality, and likelihood to send to patients. Further research is needed to optimize ChatGPT's ability to summarize radiology reports and understand what factors influence physician trust in AI-summarized reports.

Introduction

Large language models (LLMs) are a type of artificial intelligence (AI) that can simplify complex text and generate human-like responses. ChatGPT (OpenAI, San Francisco, CA) is a type of LLM that has gained widespread popularity after its release to the public in November 2022. ChatGPT is trained on vast amounts of text data from various sources including books, articles, and websites that allow it to communicate in a conversational manner. ¹ In response to a text-based prompt, ChatGPT has shown ability to compose essays, write computer code, and even pass difficult exams such as the United States Medical Licensing Exam (USMLE).^2,3 However, one of the most impressive capabilities ChatGPT has demonstrated is the ability to simplify complex text and make it more accessible for a broader audience.

One potential novel application of ChatGPT in the healthcare arena, related to this, is generating simplified radiology report summaries for patients. Following implementation of the information-blocking rule of the twenty-first Century Cures Act in April 2021, patients now have immediate and direct access to their diagnostic imaging reports. ⁴ Often times, patients access these reports before physicians are able to add interpretive notes, call to disclose results, or schedule appointments. ⁵ In oncology, these radiology reports can often include serious results such as cancer diagnoses, recurrences, and progression. Additionally, they often include complex terminology can be difficult for patients to understand. For example, one study found that only 4% of radiology reports were readable by the average US adult. ⁶

However, composition of high-quality summaries of radiology reports for patients can be time-consuming with the primary responsibility falling on physicians. With increasing physician burnout, that is partially related to the management of electronic health record in-box messages, developing tools to improve in-box message efficiency are needed. ⁷ Here, we assessed ChatGPT's ability to generate summarized MRI reports for patients with prostate cancer and evaluated the readability and physician satisfaction of an AI-summarized report. Additionally, we assessed current online patient portal response patterns from physicians at our institution.

Materials/methods

ChatGPT (GPT-3.5 series, OpenAI, San Francisco, CA) was used to summarize five unique, de-identified prostate MRI reports from patients with prostate cancer performed at our institution from 2021 to 2022 (Figure 1). A separate cohort of 50 prostate MRIs were used to assess physician online portal response patterns. The five full prostate MRIs were chosen at random. No protected health information was entered into ChatGPT to generate these reports at any time. This study qualified as exempt from requiring regulatory approval according to the Cedars-Sinai Medical Center institutional review board. Informed consent requirement was waived by the Cedars-Sinai Center Institutional Review Board. The MRI reports included different prostate cancer scenarios such as newly diagnosed, recurrent, and active surveillance. All five MRIs were summarized by two radiologists experienced in reading MRIs. Reports usually included multiple findings with different complexities. All MRIs and simplified MRI reports are included in Supplemental Figure 1.

Figure 1.

Flowchart of ChatGPT prompt and evaluation by physician questionnaire.

Prompt

Using a standardized prompt, we asked ChatGPT to summarize the full MRI reports into a patient letter at a sixth-grade reading level. The prompt was derived after multiple trials with different prompts including “Please simplify…,” “Explain like I’m a 5-year old.” The final prompt we used was “Can you please summarize the following radiology report in letter format to a patient at a sixth-grade reading level. Please include size/location of lesion and likelihood of malignancy if applicable.”

Readability

To account for variability in text output, we generated three different summarized reports per unique MRI report. We assessed if ChatGPT was able to simplify the full MRI reports at or below a sixth-grade reading level, which is the recommended readability for patient health materials per the American Medical Association (AMA) 2007 Health Literacy Manual. ⁸ Summarized reports were evaluated for readability using Flesch-Kincaid (FK) Grade Level, which is calculated using the average sentence length and average syllables per word. ⁹ FK scores of full and summarized reports were compared using Bland-Altman analysis with two-sided tests at a significance level of 0.05.

Physician assessment

Radiation oncologists at a single institution were asked to evaluate the summarized reports with an anonymous questionnaire. In the questionnaire, physicians were shown two full MRI reports that were chosen randomly and three ChatGPT-summarized versions for each full report. For each summarized report, physicians were asked six questions assessing the following: factual correctness, ease of understanding, completeness, potential for harm, overall quality, and likelihood they would send the report to a patient. A full list of survey questions is shown in Supplemental Figure 2. Qualitative responses were given on a 1 to 5 Likert-type scale ranging from strongly disagree, disagree, neither agree nor disagree, agree, and strongly agree.

Online patient portal response patterns

At our institution, physicians have the option of immediately commenting on patient imaging results through our online patient portal developed by Epic (Epic Systems Corporation, Madison, WI). Fifty consecutive prostate MRIs performed at our institution from newly diagnosed prostate cancer patients referred to our radiation oncology clinic from 2020 to 2022 were assessed to determine current physician online portal response patterns. We evaluated physician response rate to patients, time to response, and length of response via the online patient portal.

Results

Fifteen summarized reports were generated from five full MRI reports using ChatGPT. FK score for each of the five full MRI reports and 15 summarized reports is shown in Table 1. The median FK score for the full reports and summarized reports was 9.6 vs. 5.0. The median difference between FK scores for full and summarized MRI reports was 4.6 (95% CI: 4.14, 5.11, p < 0.05). The median word count for the five full MRI reports was 464 words compared to 182 words for the summarized reports (p < 0.05).

Table 1.

Readability scores for full and summarized reports.

	Report #1	Report #2	Report #3	Report #4	Report #5
Full MRI reports (n = 5)
FK score	9.9	10.3	9.5	8.8	9.2
Word count	336	464	374	566	587
Summarized reports (n = 15)
Median FK score (range)	3.8 (3.6–6.8)	5.8 (4–6.8)	5 (3.8–6)	5 (2.8–6.2)	4.8 (4.4–5.1)
Median word count (range)	187 (181–230)	125 (119–166)	182 (125–230)	219 (138–221)	190 (182–237)

Out of 18 physicians invited, a total of 12 radiation oncologists completed the questionnaire with varying levels of experience: resident (25%), attending <5 years (33%), attending 5–10 years (17%), and attending >10 years (25%). The mean [SD] ratings for all summarized reports across the six aspects evaluated in the questionnaire were as follows: factual correctness (4.0 [0.6]), ease of understanding (4.0 [0.7]), completeness (4.1 [0.5]), potential for harm (3.5 [0.9]), overall quality (3.4 [0.9]), and likelihood to send to a patient (3.1 [1.1]) (Table 2).

Table 2.

Results of physician questionnaire.

Question	Factual correctness	Ease of understanding	Completeness	Potential for harm	Overall quality	Likelihood to send to patient
All reports (n = 6)
Mean score (SD)	4.0 (0.6)	4.0 (0.7)	4.1 (0.5)	3.5. (0.9)	3.4 (0.9)	3.1 (1.1)
Summarized reports for MRI #1 (n = 3)
Mean score (SD)	4.0 (0.7)	4.3 (0.6)	4.1 (0.4)	3.5 (1.0)	3.6 (0.9)	3.1 (1.2)
Summarized reports for MRI #2 (n = 3)
Mean score (SD)	4.0 (0.7)	3.7 (0.7)	4.1 (0.6)	3.4 (0.7)	3.3 (0.9)	3.1 (1.0)

The majority of respondents agreed or strongly agreed that the summarized reports were factually correct (89%), easy to understand (78%), and complete (93%) (Figure 2). However, fewer respondents agreed or strongly agreed that the reports had low potential for harm (51%), were of high overall quality (53%), and would be sent to a patient (46%). Although less than 3% of respondents disagreed/strongly disagreed with regard to correctness, ease of understanding, and completeness; 14% and 26% of respondents disagreed/strongly disagreed with regard to overall quality and willingness to send to patient, respectively.

Figure 2.

Overall physician rating by question.

Physician responses via the online patient portal are outlined in Table 3. Overall, physicians responded to patients via the online portal for 14/50 (28%) prostate MRIs. Telephone encounters were documented in an additional 4/50 (8%) patient MRIs. The median response time was 1 day (range 0–5). Online patient portal responses by physicians were usually short, with a median word count of 28 words (range 10–76). All physician online portal responses are shown in Supplemental Figure 3.

Table 3.

Online patient portal response patterns (n = 50).

Physician responses
Online patient portal	14 (28%)
Telephone	4 (8%)
Response time
Median days (range)	1 (0–5)
Response length
Median word count (range)	28 (10–76)

Discussion

With the recent policy changes related to the twenty-first Century Cures Act, patients now have direct access to their imaging results with even high-stake results such as cancer diagnoses being released immediately. ⁴ There is now an emphasis on the need for effective communication for oncology physicians when dealing with patient portal-delivered results. ¹⁰ Waiting for oncology test results often increases patient anxiety and direct release now cuts down on wait times for receiving results by 7–14 days.^11,12 However, the ambiguity of portal-delivered results can increase anxiety, with patients now seeking information from the internet, family, and friends, which can be sources for potential misinterpretation of results. ¹³ LLMs such as ChatGPT represent a potential way to compose high-quality summaries of radiology reports for patients. While this application is promising, LLMs are not specifically trained for understanding or accuracy of statements and have the potential to generate false or misleading statements, including “hallucinations.”^14–16 Furthermore, the utility of ChatGPT in clinical-decision making capacities has yet to be demonstrated. A recent study assessing ChatGPT's performance in offering cancer treatment recommendations revealed a significant portion of recommendations partially deviated from NCCN guidelines and “hallucinations” were present in 12.5% of outputs. ¹⁷

It is necessary for physician review to validate the responses generated by ChatGPT to ensure accuracy and appropriateness.

Our study demonstrates the feasibility of using ChatGPT to generate summarized radiology reports for prostate MRIs. The AI-generated summaries significantly reduced the readability score to a sixth-grade level, making the information more accessible to the average patient. ⁸ In addition, our results indicate that current physician online portal response patterns are inconsistent, with only 28% of patients receiving a physician response to their prostate MRI at our institution. Furthermore, physician responses are often short, with a median word count of 28 words, and often do not provide patients with limited information or follow-up instructions (Supplemental Figure 3).

ChatGPT-generated summaries on the other hand have the advantage of not being limited by time, which allows it to generate more robust and detailed explanations for patients. Additionally, ChatGPT has demonstrated the ability to generate human-like and empathetic responses. A recent study demonstrated that ChatGPT chatbots exhibited almost a 10 times higher proportion of empathetic vs very empathetic responses compared to physicians in addressing patient questions. ¹⁴ Other recent studies have also demonstrated the potential utility of LLMs to generate after visit instructions, follow-ups, and summarizing medical records.^18–20

In a recent multi-site survey of current patient online portal preferences, of 8139 patients surveyed, the majority (96%) expressed a preference for receiving test results through the online patient portal immediately upon reporting and before being contacted by a health care practitioner. ²¹ Most respondents reported that reviewing the results had either a positive effect or no effect on their level of worry. However, in a subset of respondents with abnormal results, they experienced additional worry. Despite this, the overwhelming majority (95%) would still rather continue to receive immediately released results through the portal. Other studies have also demonstrated that patients receiving abnormal results are at increased risk for negative emotions, often due to difficulty in interpreting the results in the context of their own health.^22,23 ChatGPT-generated summaries can potentially alleviate some of this worry by providing clearer and more informative explanations of the results. Additionally, by integrating AI-generated summaries into the patient portal, clinicians can better focus on discussing the implications of the results and addressing patient concerns.

In our study, we found that ChatGPT demonstrated an excellent capability to summarize multiple findings in a report and generate a concise, clear, and comprehensive report. Indeed, the majority of physicians found the AI-summarized reports in our study were factually correct, easy to understand, and complete. However, there were concerns regarding potential for harm, overall quality, and likelihood of sending the reports to patients. These concerns may be attributed to several factors that influence physician trust in AI-summarized reports. Firstly, physicians may be hesitant to rely on AI-generated summaries due to concerns about potential inaccuracies, misinterpretations, or omissions of important information.^24,25 Additionally, a known limitation of LLMs like ChatGPT is the potential for “hallucinations,” where the model includes findings and information that were not included in the original report.^26,27 In our study, we noticed differences between AI-generated summaries regarding length, level of detail, and content even when given the same prompt. Random results such as these are inherent to LLMs, but can decrease physician trust in AI-generated reports. In a recent study evaluating feasibility of ChatGPT to summarize radiology reports, the authors found that surveyed radiologists were satisfied with the correctness and completeness of ChatGPT-generated reports. ²⁸ However, 51% of radiologists found incorrect passages in the summaries and instances of missed key medical findings, imprecise language, and potential for misinterpretation. Currently, ChatGPT does not have a built-in template for its summarized report generation. In the future, providing ChatGPT with a consistent template with clear instructions for formatting may help ChatGPT generate more reliable and accurate radiology report summaries.

ChatGPT has already demonstrated the potential of LLMs in healthcare with multiple applications being explored ranging from documentation, clinical practice support, education, remote patient monitoring, and medication management to name a few. ²⁹ Although ChatGPT offers significant promise, extensive research will be needed to understand how to best deploy LLMs and address concerns before its widespread implementation. This study emphasizes the need for more efficient communication through online patient portals and potential of AI-generated summaries in addressing challenges faced by both patients and physicians. However, the acceptability of this technology from a patient perspective must be considered, as concerns of automation and depersonalization of care could lead to resistance in adopting this technology. ²⁰ Future research should focus on eliciting patient perspectives and exploring factors that influence physician trust in AI-generated radiology report summaries.

Our study has several limitations including relatively small sample size and a single institution study, which may limit the generalizability of our findings. Currently, we do not envision this technology being used without human interaction and further large sample research will be needed to validate the use of LLMs to summarize radiology reports. Second, we focused exclusively on prostate cancer MRI reports, and the results may not be directly applicable to other types of radiology reports or cancer diagnoses. Third, the anonymous questionnaire used in our study may be subject to response bias, as physicians with strong opinions on AI-generated summaries may be more likely to participate. Additionally, our study found that despite high ratings regarding summaries being “factually correct, easy to understand, and complete” there were concerns regarding “potential for harm, overall quality, and likelihood of sending the report to patients.” Our study is unable to identify potential causes for this discrepancy, and it is beyond the current scope of our study. Finally, it's possible the physicians communicated results with patients and did not document this in the electronic medical record.

Conclusion

In conclusion, this study demonstrates the novel feasibility of using ChatGPT to generate patient-friendly summaries of prostate cancer MRI reports, significantly reducing the readability score to a sixth-grade level. This has the potential to make radiology reports more accessible and understandable for patients, particularly in the context of online patient portals, where patients often receive results directly without physician guidance. The majority of physicians agreed that AI-generated summaries were factually correct, easy to understand, and complete. However, there were concerns regarding potential for harm, overall quality, and willingness to send the reports to patients. Further studies will be needed to understand patient perspectives as well as factors that influence physician trust in AI-generated summaries.

Supplemental Material

sj-docx-1-dhj-10.1177_20552076231221620 - Supplemental material for Feasibility and acceptability of ChatGPT generated radiology report summaries for cancer patients

Supplemental material, sj-docx-1-dhj-10.1177_20552076231221620 for Feasibility and acceptability of ChatGPT generated radiology report summaries for cancer patients by Eric M Chung, Samuel C Zhang, Anthony T Nguyen and Katelyn M Atkins, Howard M Sandler, Mitchell Kamrava in DIGITAL HEALTH