Suitability of ChatGPT as a Source of Patient Information for Screening Mammography

Abstract

ChatGPT3.5 and ChatGPT4 were released publicly in late November 2022 and March 2023, respectively, and have emerged as convenient sources of patient health education and information, including for screening mammography. ChatGPT4 offers enhanced capabilities; however, it is only available by paid subscription. The purported benefits of ChatGPT for health education need to be objectively evaluated. To assess performance differences, ChatGPT3.5 and GPT4 were used between 13 April and 29 May 2023 to generate breast screening patient information sheets, which were evaluated using the Patient Education Materials Assessment Tool for printed materials (PEMAT-P) and the CDC Clear Communication Index (CDC Index) Score Sheet; and benchmarked against gold standard content in BreastScreen NSW’s patient information sheet. Mean scores were reported for comparison. GPT3.5 provided the appropriate tone and currency of information but lacked accuracy, omitting key insights: PEMAT-P understandability 68.0% (SD = 6.56) and actionability 36.7% (SD=20.4); CDC Index 58.8% (SD = 15.3). GPT4 was deemed superior to GPT3.5 but included several key omissions: PEMAT-P understandability 75.0% (SD = 17) and actionability 53.3% (SD = 11.54); CDC Index 66.0% (SD = 4.1). Both ChatGPT versions exhibited poor understandability and actionability and were unclear in their messaging. Those with poor health literacy will not benefit from accessing current versions of ChatGPT and may be further disadvantaged if they do not have access to a paid subscription. ChatGPT is evidenced to be an unreliable and inaccurate source of information concerning breast screening that may undermine participation and risk increased morbidity and mortality from breast cancer. ChatGPT may increase the demand on health care educators to rectify misinformation.

Background

In Australia, breast cancer affects one in seven women and is the second most common cause of female mortality (National Breast Cancer Foundation, 2023). When detected in its pre-clinical stage, breast cancer mortality and morbidity are reduced. The gold standard diagnostic imaging test for breast cancer is mammography, with women aged between 50 and 74 years offered free biannual screening through BreastScreen Australia sites. Over many years, patient information sheets have been developed by BreastScreen NSW to educate women about the risks of breast cancer and the benefits of screening mammography and to support informed consent for the procedure (BreastScreen NSW, 2023). As a multicultural country, information sheets are published in 28 community languages, all aimed at evoking participation in screening by enabling informed decision-making.

ChatGPT is a generative pre-trained transformer (GPT) large language model by OpenAI, trained to process and generate human-like text (Biswas, 2023). The artificial intelligence (AI) chatbot was made publicly available on 30 November 2022. The GPT3.5 architecture has captured significant interest, including in health, where it has been used or proposed to offer benefits in medical writing and provide health education (Biswas, 2023). At the time of writing, there was a paucity of scientific evaluations of the accuracy, appropriateness, clarity, and currency of ChatGPT medical writing targeted at professionals and patients. It has been found that both ChatGPT4 and ChatGPT3.5 exhibited a moderate level of accuracy in determining the appropriate imaging steps for breast cancer screening and the assessment of breast pain (Rao et al., 2023). Most research on ChatGPT refers specifically to GPT3.5, and most discussions of ChatGPT include a glance forward at the educational possibilities presented by the more advanced GPT4. GPT4 was released as a paid subscription in March 2023, although some of the features were not yet enabled (e.g., image inputs); GPT4 is also constrained by the September 2021 learning cut-off. However, GPT4 problem-solving and accuracy are reported to be superior to GPT3.5 by 60% (Sanderson, 2023).

However, to be useful, health education information must be suitable for many different audiences with variable health literacies and contribute to public health and positive health behaviors, including participation in breast screening (Baur & Prue, 2014).

The internet is a well-established source of health information, with as many as 80% of internet users searching for health and medical information (Azer et al., 2017). Search engines can return various sources with incorrect or misleading information, leaving the patient to navigate and determine what advice to accept as real. These internet resources often lack quality and patient readability and create confusion (Roughead et al., 2016). Importantly, deciphering information among numerous web pages returned for any search requires a high degree of health literacy. Indeed, the accessibility of health and health information along socioeconomic lines can be exacerbated in the digital space due to variations in health literacy (Smith & Magnani, 2019). Generative AI like ChatGPT can address health literacy limits by responding consistently to specific inquiries. This assumes that the nature of the query is within the training set of the AI tool. A recent comparison of internet search engines with generative AI reveals the superiority of ChatGPT over Google for medical knowledge but inferiority for medical recommendations (Ayoub et al., 2024). In some ways, this defines the purpose of ChatGPT, which itself reminds patients that individual medical advice and diagnosis are not the remit of ChatGPT and that such advice and recommendations should be sought from an expert health practitioner. ChatGPT should not be used to replace clinical decision-making. In health care practice, concerns have also been raised about the data sets applied in ChatGPT training (Deng & Lin, 2023; Sallam, 2023). These concerns highlight the potential for bias, which may constrain ChatGPT capabilities and knowledge and give rise to factual inaccuracies, unlike dedicated breast screening promotional material that ensures quality-controlled information. Between 2020 and 2022, BreastScreen Australia’s participation rate for eligible women was only 48% compared to the program’s 70% target (Australian Institute of Health and Welfare, 2021). Misinformation can cause uncertainty among women, which can impact their willingness to participate in screening programs. This, in turn, results in missed opportunities for the early detection of breast cancer, potentially resulting in delayed diagnoses and poorer prognoses, including increased mortality rates.

Purpose

This research aims to evaluate the capability of ChatGPT to generate suitable patient information concerning screening mammography that is fit for purpose, including the understandability and actionability of the material provided. Patient information relating to screening mammography plays an important role in educating patients with sufficient insight so they can make an informed decision regarding breast cancer risk and the benefits of screening; ideally such information should prompt positive health behavior and participation in breast screening. ChatGPT presents as an alternative to traditional health promotion practice that warrants investigation. The secondary aims were to compare the accuracy of information provided between GPT3.5 and GPT4 and to benchmark ChatGPT information against the gold standard BreastScreen NSW health promotion material.

Method

A Conversation With ChatGPT

To evaluate the capabilities of ChatGPT for generating patient information concerning screening mammography, ChatGPT was asked to generate a patient information sheet for screening mammography, and separate conversations were started with both versions of ChatGPT between 13 April and 29 May 2023. The following seven questions derived from key messaging in a BreastScreen NSW benchmark exemplar (BreastScreen NSW, 2023) were used to establish the context of the conversation: Question 1: What is mammography?; Question 2: What is the difference between screening mammography and diagnostic mammography?; Question 3: Is screening mammography effective in saving lives from breast cancer?; Question 4: Who should have screening mammography?; Question 5: What about breast implants, breast surgery, dense breasts, and breast radiation therapy?; Question 6: How does a mammogram manage implantable devices like an implantable cardiac device (ICD)? Question 7: Who should avoid having mammography?

ChatGPT3.5 and 4 were then asked the following questions: Question 8: Can you write a patient information sheet suitable for providing sufficient information that it could be used to gain informed consent for a screening mammogram?; Question 9: Please provide references and citations that support your claims of safety and benefit.; Question 10: But what about scientific research and trials published in peer-reviewed journals?

BreastScreen NSW Benchmark

An online search was undertaken for the latest versions of publicly available material published by BreastScreen NSW relating to the breast screening procedure. The patient information sheet Breast Screening and You (BreastScreen NSW, 2023) was selected by author consensus as best aligned to the task and used as a benchmark against the content of the ChatGPT information sheets. The Australian government-funded publication provides accurate and up-to-date information on screening designed to maximize participation rates and is a known and credible source of information. The publication Breast Screening and You (BreastScreen NSW, 2023) has been designed to accommodate all health literacies, promote understanding of breast cancer and the benefits of screening and its process as well as to have a message that is understandable and actionable.

Evaluating ChatGPT

The Patient Education Materials Assessment Tool for printed materials (PEMAT-P) patient information sheet (Shoemaker et al., 2014) was used to assess the ChatGPT 3.5 and ChatGPT4.0 responses and the BreastScreen NSW information sheet. The PEAMAT-P has previously been used in Australia to assess health promotion materials and is validated as a tool for use by both professionals and the general public, including those with low health literacy (Mastroianni et al., 2019; Shoemaker et al., 2014). The PEMAT-P is considered an effective supplementary tool to reading alone (Vishnevetsky et al., 2018). Each ChatGPT information sheet was reviewed by members of the expert panel, who have clinical experience in breast screening and have published extensively on the topic. The overall score for each material content category was averaged to calculate mean scores for each category, with standard deviation.

The CDC Clear Communication Index was also used to assess each information sheet to further assess the language used in each communication. The index assesses seven key areas of communication—the main message or call to action, use of plain language, information design, state of science, behavioral recommendations, numbers, and risk. These criteria are based on the index used to develop and assess communication for diverse audiences (Baur & Prue, 2014).

The authors compared the content of the three information sheets and recorded key omissions and misinformation.

The references provided were analyzed for accuracy.

Results

The conversations with ChatGPT3.5 and ChatGPT4 in response to questions one to six are reported in Table 1.

Table 1

Chat With ChatGPT3.5 and ChatGPT4

Question 1: What is mammography?
ChatGPT3.5: Mammography is a medical imaging technique that uses low-dose X-rays to create images of the breast. It is primarily used for screening and detecting breast cancer in women who do not have any symptoms, as well as for diagnosing breast cancer in women who have symptoms such as a lump or breast pain. During a mammogram, the breast is compressed between two plates and a small amount of radiation is used to take an X-ray image. The resulting image can show any abnormal changes in the breast tissue, such as lumps, calcifications, or masses. These changes can sometimes indicate the presence of cancerous cells. Mammography is an important tool in the early detection of breast cancer, and it is recommended that women over the age of 50 get a mammogram every two years. Women with a family history of breast cancer or other risk factors may need to start screening at an earlier age or get screened more frequently.	ChatGPT4: Mammography is a type of imaging that uses low-dose X-rays to examine the breasts. It’s a crucial tool in early detection of breast cancers, as it can show changes in the breast up to two years before a patient or physician can feel them. There are two types of mammography: 1. Screening Mammography: This is a routine procedure for women who have no symptoms of breast cancer. It aids in detecting cancer early, thus improving the chances for successful treatment. 2. Diagnostic Mammography: This is used after suspicious results on a screening mammogram or after some signs of breast disease are found. It could include a lump, breast pain, nipple discharge, thickening of skin on the breast, or changes in the size or shape of the breast. While mammography is an effective screening tool, it’s not perfect. Some cancers may not be detected by a mammogram, while in other cases, the mammogram may suggest an abnormality that, upon further testing, turns out to be non-cancerous. It’s just one tool among others, like ultrasound and MRI, that can be used for the detection of breast cancer. As with any medical procedure, the risks and benefits should be discussed with your healthcare provider.
Question 2: What is the difference between screening mammography and diagnostic mammography?
ChatGPT3.5: Screening mammography and diagnostic mammography are two types of mammograms that serve different purposes. Screening mammography is a routine test used to detect breast cancer in women who do not have any symptoms. It is typically recommended for women over the age of 50 or for those who have a higher risk of developing breast cancer due to factors such as family history or genetic mutations. During a screening mammogram, X-ray images of both breasts are taken from different angles. The images are then examined by a radiologist to check for any signs of breast cancer. Diagnostic mammography, on the other hand, is used to investigate breast abnormalities that have been detected during a physical exam, screening mammogram, or other breast imaging test. Diagnostic mammography involves taking additional images of the breast from different angles and with a higher level of detail. The images are then carefully evaluated by a radiologist to determine the nature of the abnormality, such as whether it is a benign cyst or a cancerous tumour. In summary, screening mammography is a routine test used to detect breast cancer in asymptomatic women, while diagnostic mammography is used to investigate abnormalities in the breast and to aid in the diagnosis of breast cancer.	ChatGPT4: Both screening and diagnostic mammography are used for examining the breasts using X-rays, but they are used in different situations and serve different purposes. Screening mammography is a routine procedure used to detect breast cancers in women who are asymptomatic, i.e., they have no signs or symptoms of the disease. The goal is early detection, before physical symptoms develop, as treatment is often more successful at this stage. Typically, two X-ray images of each breast are taken – one from a side view and one from above. Diagnostic mammography is used when a woman has breast symptoms or when a change is detected on a screening mammogram. These symptoms may include a lump, pain, nipple thickening or discharge, or a change in breast size or shape. In this case, more detailed X-rays are taken than during screening mammography. The radiologist may zoom in on a particular area of concern, take images from additional angles, or use special techniques to compress or magnify the breast tissue. In summary, the main differences between the two types of mammography are their purpose (screening for early detection in asymptomatic women versus diagnosing a known or suspected problem) and the extent of the examination (routine views versus more detailed and targeted imaging).
Question 3: Is screening mammography effective in saving lives from breast cancer?
ChatGPT3.5: Yes, screening mammography has been shown to be effective in saving lives from breast cancer. Screening mammography can detect breast cancer at an early stage, before it has a chance to grow and spread, making treatment more effective and increasing the chances of survival. Studies have shown that regular screening mammography can reduce the risk of dying from breast cancer by up to 40%. The earlier breast cancer is detected, the more treatment options are available, and the better the chances of survival. However, it’s important to note that mammography is not perfect and has some limitations. False positives (when a mammogram suggests there is cancer when there is not) and false negatives (when a mammogram misses a cancer that is present) can occur. Additionally, some breast cancers may not be detected by mammography, especially in women with dense breast tissue. Therefore, mammography is just one tool in the fight against breast cancer, and women should work with their healthcare providers to determine their individual risk factors and appropriate screening plan.	ChatGPT4: Screening mammography has been shown to reduce the mortality rate from breast cancer, primarily because it can detect tumors at an earlier stage when they’re more treatable. Earlier detection can allow for less aggressive treatment options and improved chances of survival. According to the American Cancer Society, mammograms have helped reduce breast cancer mortality in the United States by nearly 40% since 1990. Moreover, the World Health Organization also suggests that when screening mammography is used appropriately, it can reduce breast cancer mortality by around 20%. However, it’s important to note that while mammography is a valuable tool for early detection, it’s not perfect. It has some limitations and potential risks: • False Positives: Mammograms can sometimes indicate a problem when there isn’t one (known as a “false positive”), leading to unnecessary stress, additional testing, and even unnecessary treatment. • False Negatives: In some cases, mammograms can miss detecting breast cancer (known as a “false negative”). This is more common in younger women who typically have denser breast tissue. • Overdiagnosis and Overtreatment: Some of the cancers detected through screening mammograms are slow-growing and may never cause symptoms or threaten a woman’s life. However, it’s currently impossible to predict which cancers will remain harmless, so all detected cancers are generally treated, leading to a situation of potential overdiagnosis and overtreatment. • Radiation Exposure: While the amount of radiation used in modern mammography is quite low, repeated X-rays do have the potential to cause cancer. The risk is higher for younger women. In general, the decision to start screening mammography should be an individual one, taking into account a woman’s personal risk factors, the potential benefits, and the potential harms. It’s a conversation that each woman should have with her healthcare provider.
Question 4: Who should have screening mammography?
ChatGPT3.5: The decision to have screening mammography should be made on an individual basis, taking into account a woman’s age, risk factors, and personal preferences. However, there are some general guidelines regarding who should have screening mammography. In the United States, the American Cancer Society recommends that women with an average risk of breast cancer should start having screening mammograms at age 45, and continue every year until age 54. After age 55, women can continue to have mammograms every year or switch to having them every two years, depending on their personal preferences and risk factors. Women with a higher risk of breast cancer due to factors such as a family history of breast or ovarian cancer, genetic mutations, or previous breast biopsies may need to start having mammograms at an earlier age or more frequently. In these cases, a healthcare provider may recommend starting screening as early as age 30 or 40 and having mammograms every year or every other year. It’s important for women to talk to their healthcare provider about their individual risk factors and to develop a personalized screening plan based on their age, family history, and other factors.	ChatGPT4: Screening mammography is typically recommended for women aged 40 and older, although the exact age and frequency depend on guidelines from different organizations and personal risk factors. Here are a few general guidelines from prominent health organizations as of my last update in 2021: • American Cancer Society: Recommends women with an average risk of breast cancer should begin annual mammograms at age 45, and can switch to every 2 years (or continue annually) starting at age 55. However, women should have the opportunity to start annual screening as early as age 40 if they choose to do so. • U.S. Preventive Services Task Force: Recommends biennial screening mammography for women aged 50 to 74 years. The decision to start screening mammography in women prior to age 50 years should be an individual one. • National Comprehensive Cancer Network: Recommends annual mammograms beginning at age 40 for women with an average risk of breast cancer. Individual risk factors can modify these recommendations. For example, women with a personal or family history of breast cancer, a genetic tendency, or certain other risk factors, should work with their healthcare providers to develop an individualized screening plan which might involve starting screening earlier, having additional tests, or having more frequent exams. Given the ongoing debate about the best age to start and how often to have mammograms, women should discuss the potential benefits, harms, their personal risk factors, and their personal preferences with their healthcare providers to make a shared decision that best fits their needs and circumstances. Keep in mind that guidelines may have changed if you’re asking this question after my last training data in September 2021, and you should always follow the most recent guidelines or your healthcare provider’s advice.
Question 5: What about breast implants, breast surgery, dense breasts and breast radiation therapy?
ChatGPT3.5: Breast implants, breast surgery, dense breasts, and breast radiation therapy can all affect the accuracy of mammography and may require additional imaging or special techniques. For women with breast implants, mammography can be more difficult to perform and may require additional views or techniques to properly visualize the breast tissue. Women with breast implants should inform the mammography facility about their implants and the type of surgery they had, as this can affect the way the mammogram is performed and interpreted. Breast surgery, such as a lumpectomy or mastectomy, can also affect the accuracy of mammography. Women who have had breast surgery should inform the mammography facility and their healthcare provider about their surgery, as this may affect the need for additional imaging or special techniques. Dense breast tissue can also make it more difficult to detect breast cancer on a mammogram. Women with dense breast tissue may need additional imaging such as ultrasound or magnetic resonance imaging (MRI) to supplement their mammography. Breast radiation therapy can cause changes in the breast tissue that may make it more difficult to interpret mammography results. Women who have had breast radiation therapy should inform the mammography facility and their healthcare provider about their treatment, as this may affect the need for additional imaging or special techniques. In summary, women with breast implants, breast surgery, dense breasts, or breast radiation therapy may need additional imaging or special techniques to ensure the accuracy of their mammography results. It’s important to inform the mammography facility and healthcare provider about any breast-related procedures or treatments to ensure appropriate screening and follow-up.	ChatGPT4: All the factors you mentioned - breast implants, breast surgery, breast density, and radiation therapy to the breast - can impact breast cancer screening. Here’s how: • Breast Implants: Mammography can be less accurate in women who have breast implants because the implants can obscure some of the breast tissue. Special techniques called displacement views can be used to push the implant out of the way and image as much of the natural breast tissue as possible. However, even with these techniques, some breast tissue may still be hard to see. Women with implants should let the mammography facility know about their implants when they schedule their mammogram. • Breast Surgery: Depending on the type and extent of the surgery, this can sometimes make mammograms harder to interpret. If you’ve had surgery, it’s important to choose a mammography center with technologists and radiologists who are experienced in x-raying breasts after surgery. • Dense Breasts: Dense breast tissue can make it harder to interpret a mammogram since cancer can be hidden by the dense tissue, as both appear white on the mammogram. Dense breasts can also increase the risk of breast cancer. Additional screening methods like ultrasound or MRI may be recommended for women with dense breasts. • Radiation Therapy: If you’ve had radiation therapy to the chest before the age of 30, your risk of breast cancer is increased, and your healthcare provider might recommend additional or earlier screening. In all of these cases, it’s important to have a conversation with your healthcare provider about your personal risk and the best screening strategy for you. These factors don’t necessarily preclude you from having a mammogram, but they may mean you need to have additional or different types of screening tests.
Question 6: How does a mammogram manage implantable devices like an implantable cardiac device (ICD)?
ChatGPT3.5: During a mammogram, the breast is compressed between two plates and X-rays are used to image the breast tissue. While this process is generally safe for most women, it may not be suitable for women with certain types of implantable medical devices, such as implantable cardioverter-defibrillators (ICDs). ICDs are implanted devices that can help regulate heart rhythm and prevent sudden cardiac arrest. The devices contain a battery, electrical leads, and other components that can be affected by the electromagnetic fields generated by the mammography equipment. To avoid potential interference with the ICD, special precautions may need to be taken during the mammogram. The woman’s healthcare provider and the radiology staff should be informed of the presence of the ICD before the mammogram. Depending on the type and location of the ICD, the mammography staff may need to adjust the positioning of the breast or use shielding to protect the ICD from the X-rays. In some cases, the woman’s healthcare provider may recommend an alternative imaging modality, such as ultrasound or MRI, to avoid potential interference with the ICD. It’s important for women with implantable medical devices to discuss any concerns they may have about mammography or other imaging modalities with their healthcare provider, who can provide individualized recommendations based on the woman’s specific medical history and device characteristics.	ChatGPT4: An Implantable Cardioverter Defibrillator (ICD) is a device placed under the skin of the chest or abdomen to help treat irregular heartbeats, known as arrhythmias. Although an ICD is not placed in the breast tissue, it’s important to inform the technologist performing the mammogram if you have one. Here’s why: • The technologist should know about all implantable devices and health conditions to take the best care of you during the procedure. • An ICD contains metal and can interfere with the X-ray image on a mammogram. The technologist can adjust their technique to ensure they get the best images of your breast tissue. • The technologist can also make sure that the compression during the mammogram doesn’t put too much pressure on the device. • Although it’s extremely rare, there’s a theoretical risk that the ICD could interpret the mammogram machine’s electrical signals as a heart rhythm problem. However, this would be highly unusual and unlikely. In summary, having an ICD should not prevent you from getting a mammogram. But as with any medical condition or device, you should inform the mammography technologist beforehand. As always, if you have any questions or concerns, you should discuss them with your healthcare provider.
Question 7: Who should avoid having mammography?
ChatGPT3.5: Most women can safely undergo mammography, but there are some situations where mammography may not be appropriate or may need to be delayed. Women who are pregnant or breastfeeding are generally advised to postpone mammography until after delivery or after breastfeeding is complete, as the breast tissue can be more sensitive during these times. Women with a known allergy to contrast material (dye) or to the materials used in mammography equipment should inform the mammography facility, as special precautions may need to be taken to ensure their safety during the exam. Women with breast implants should also inform the mammography facility, as additional imaging or special techniques may be necessary to properly visualize the breast tissue. If a woman has had a recent mammogram and is experiencing breast symptoms, such as a lump or nipple discharge, she may need additional imaging, such as ultrasound or MRI, rather than a repeat mammogram. In general, it’s important for women to discuss their individual risk factors and any concerns they may have about mammography with their healthcare provider. Based on this discussion, the healthcare provider can recommend the appropriate screening plan and schedule for each woman.	ChatGPT4: In general, mammography is a well-tolerated procedure, but there are certain circumstances or conditions when it might not be recommended or additional considerations should be taken: • Pregnancy: The amount of radiation from a mammogram is low, but because there’s a potential risk to the fetus, mammograms are generally avoided during pregnancy unless absolutely necessary. If a pregnant woman has a suspicious lump or other worrying symptoms, other types of imaging such as ultrasound may be used. • Young Age: Women under the age of 30 rarely get mammograms. Their breast tissue is typically dense, which can make mammograms less effective. Also, because breast cancer is rare in this age group and the risk of radiation exposure is greater, other imaging methods like ultrasound or magnetic resonance imaging (MRI) might be used if necessary. • Women With Low Risk and No Symptoms: The potential benefits of screening mammography may not outweigh the risks for women who are at very low risk for breast cancer and have no symptoms. This might include women who are older and have multiple serious health conditions that would make treatment for breast cancer risky or of little benefit. • Women with Certain Breast Implants: While mammography can be performed in women with breast implants, care must be taken to ensure that the implant is not damaged during the procedure. Sometimes, additional imaging methods may be required for a full evaluation. • Previous High Dose Radiation Therapy: Women who received high-dose radiation therapy, especially at a young age, may be at an increased risk of developing radiation-induced breast cancer. They should discuss this with their healthcare provider. Remember, individual healthcare decisions should be made in consultation with a healthcare provider who can consider the patient’s overall health, risk factors, and preferences. It’s important to discuss any concerns about mammography with your healthcare provider to make the best decision for your circumstances.

The patient information sheets generated by GPT3.5 and GPT4 responses are reported in Figures 1 and 2, respectively.

Figure 1

GPT3.5 Generated Patient Information Sheet

Figure 2

GPT4 Generated Patient Information Sheet

When asked to provide references and citations that supported claims of safety and benefit in the patient information sheet, Chat 3.5 responded:

“Sure, here are some references and citations to support the claims made in the patient information sheet” and provided the following references which are reported here verbatim: The American Cancer Society. (2021). Breast cancer early detection and diagnosis. Retrieved from https://www.cancer.org/cancer/breast-cancer/screening-tests-and-early-detection/mammograms.html; National Cancer Institute. (2021). Mammograms. Retrieved from https://www.cancer.gov/types/breast/mammograms-fact-sheet; American College of Radiology. (2021). Mammography. Retrieved from https://www.acr.org/Clinical-Resources/Radiology-Safety/Mammography, and National Breast Cancer Foundation. (2021). Mammogram. Retrieved from https://www.nationalbreastcancer.org/mammogram/

Chat GPT3.5 further explained that:

These sources provide evidence that mammography screening is a safe and effective tool for early detection of breast cancer, with the potential to reduce breast cancer mortality by up to 40% in women over the age of 50. The risk of radiation exposure during mammography is low, and the benefits of early detection and treatment of breast cancer generally outweigh any potential risks. However, it is important to discuss any concerns or questions with a healthcare provider.

ChatGPT4 automatically included the following references in the original response to develop a patient information sheet which are reported here verbatim: American Cancer Society. (2019). Mammograms. Retrieved from https://www.cancer.org and U.S. Food & Drug Administration (FDA). (2020). Mammograms. Retrieved from https://www.fda.gov

When asked about scientific research and trials published in peer-reviewed journals, Chat GPT3.5 responded:

“Certainly, here are some references to scientific research and trials published in peer-reviewed journals that support the safety and benefit of mammography” and provided the following references which are reported here verbatim: Duffy, S. W., et al. (2020). Effect of mammographic screening from age 40 years on breast cancer mortality at 10 years’ follow-up: a randomised controlled trial; The Lancet, 395(10242), 371–380; Nelson et al. (2016). Screening for breast cancer: An updated systematic review and meta-analysis. JAMA, 315(13), 1404–1414; Kerlikowske, K., et al. (2015). Identifying women with dense breasts at high risk for interval cancer: a cohort study. Annals of Internal Medicine, 162(10), 673–681 and Bleyer, A., et al. (2016). Effectiveness of screening mammography among women aged 40 to 49 years and 50 to 69 years: comparison of relative and absolute benefit. JAMA Internal Medicine, 176(10), 1541–1548.

GPT4 replied:

“Sure, here are some specific references to the studies and guidelines I mentioned ” and provided the following references which are reported here verbatim: Tabár L, Yen MF, Vitak B, Chen HH, Smith RA, Duffy SW. Mammography service screening and mortality in breast cancer patients: 20-year follow-up before and after introduction of screening. Lancet. 2003 Apr 26;361(9367):1405–10; Miller AB, Wall C, Baines CJ, Sun P, To T, Narod SA. Twenty five year follow-up for breast cancer incidence and mortality of the Canadian National Breast Screening Study: randomised screening trial. BMJ. 2014 Feb 11;348:366; Siu AL; U.S. Preventive Services Task Force. Screening for Breast Cancer: U.S. Preventive Services Task Force Recommendation Statement. Ann Intern Med. 2016 Feb 16;164(4):279–96 and Gøtzsche PC, Jørgensen KJ. Screening for breast cancer with mammography. Cochrane Database Syst Rev. 2013 Jun 4;(6):CD001877.

References from the BreastScreen NSW patient information sheet were located at the end of the publication (BreastScreen NSW, 2023).

The PEMAT-P and CDC Index scores are presented in Table 2 (BreastScreen NSW, 2023). ChatGPT4.0 was scored higher by the expert panel than ChatGPT3.5 for understandability, actionability, and clear communication.

Table 2

PEMAT-P and CDC Index Scores

Information sheet source	ChatGPT3.5				ChatGPT4.0
PEMAT Reviewer	1	2	3	Mean	1	2	3	Mean
Understandability Score	62.0%	75.0%	67.0%	68.0% SD = 6.56	75.0%	92.0%	58.0%	75.0% SD = 17
Actionability Score	33.0%	20.0%	60.0%	36.7% SD = 20.4	40.0%	60.0%	60.0%	53.3% SD = 11.54
CDC Clear Communication Index Score** Reviewer	1	2	3		1	2	3
Part A: Core	4/11	4/11	8/11		6/11	7/11	7/11
Part B: Behavioral Recommendations	2/3	3/3	3/3		2/3	3/3	3/3
Part C: Numbers	0/3 (N/A)	1/3	0/3 (N/A)		0/3 (N/A)	1/3	0/3 (N/A)
Part D: Risk	2/2	2/3	2/3		2/2	2/3	2/3
Total	50.0%	50.0%	76.5%	58.8% SD = 15.3	62.5.0%	65.0%	70.6%	66.0% SD = 4.15

Note. PEMAT-P: Interpret the PEMAT scores. The higher the score, the more understandable or actionable the material. CDC: If the total score is 90 or above: most items that make materials easier to understand and use have been addressed.(Shoemaker et al., 2014).

In this study, the first citation for GPT3.5 is not real. The citation appears to hybridize two articles (Duffy et al., 2020; Moss et al., 2006). The second citation for GPT3.5 also does not exist and could be a mix of several articles (Nelson, Cantor, et al., 2016; Nelson, Fu, et al., 2016). The last GPT3.5 citation is also fabricated, presumably from a hybridisation (Bleyer et al., 2016; Kerlikowske, 1997). The third link provided by GPT3.5 is not valid. Neither of the GPT4 references is correct, although both organizations will provide information on mammograms. The references used in the BreastScreen NSW patient information sheet are accurate as cited.

Discussion

Educational health behavior change campaigns to increase participation in the BreastScreen Australia breast screening program are regularly made accessible across mass media. In addition, print material is also available for women to take home to read or review at their appointments, including the BreastScreen NSW patient information sheet used for benchmarking in this study. The clinically available patient information sheet Breast Screening and You (BreastScreen NSW, 2023) was used to critique the ChatGPT-generated versions of this study. GPT3.5 provided good general information but lacked the detail required to gain informed consent. This may be because ChatGPT has not had the necessary training to achieve reproducibility of information at acceptable standards. Explaining the procedure itself, for example, was correct at a general level but missed key details that a patient should expect to understand before progressing with the procedure, which has the potential to discourage participation. The responses for both GPT3.5 and GPT4, as noted by the authors, reflect a U.S.-centric bias related to a ChatGPT construct which can produce misleading or incorrect information in an alternate setting, such as BreastScreen NSW in this instance. Some notable omissions from the GPT3.5 patient information sheet included not mentioning a family history of breast cancer or that most women with breast cancer (90%) do not have a family history; that breast cancer is a common disease or that it increases with age; breast cancer risk factors other than family history or genetic mutations; a screening mammogram can detect small breast cancers before symptoms are present; that a referral is not required; and requirements for imaging pregnancy or breastfeeding patients.

While ChatGPT3.5 provided details of screening and patient age, the information was considered U.S.-centric and, therefore, incorrect in the Australian context. The information provided did not outline that screening mammography is for women without symptoms or direct women with symptoms to the appropriate medical pathway; it also did not suggest wearing a two-piece outfit so the top can be removed without removing the bottom despite indicating that patients would be in a gown from the waist up. The procedure was not adequately explained, nor were the two projections taken and implied positioning was automated. The information also did not indicate the necessity to advise staff in advance if they have implants, implantable devices, a history of breast surgery or a history of breast irradiation. Advice for women under the age of 40 or for those with dense breasts was not provided. The frequency of screening or when the next mammogram would be needed was not explained. Mammogram limitations, including false positives and false negatives, were not explained, and while mentioning potential discomfort, it did not indicate the procedure could be stopped in a small portion of women where the procedure is too painful. This misinformation may dissuade participation.

Like GPT3.5, the GPT4 patient information sheet omitted mentioning: a family history of breast cancer or that most women with breast cancer (90%) do not have a family history; indicating breast cancer is a common disease or that it increases with age; breast cancer risk factors other than family history or genetic mutations; that a referral is not required; directing women with symptoms to the appropriate medical pathway; a suggestion to wear a two-piece outfit so the top can be removed without removing the bottom despite indicating that patients would be in a gown from the waist up. While mentioning potential discomfort, it did not indicate that the procedure could be stopped in a small portion of women where the procedure is too painful; advice for women under the age of 40 years or those with dense breasts; indication of the frequency of screening or when the next mammogram would be needed. The information provided did not explain the limitations of mammography, including false positives and false negatives; it did not mention requirements for pregnancy or breastfeeding patients; it did not adequately explain the procedure or the two projections taken.

GPT4 provided some information not evident in the GPT3.5 version, including advice to inform staff about breast implants and that screening mammograms can detect small breast cancers before symptoms are present. GPT4 did, however, omit some important pieces of information that the GPT3.5 version had included. This information includes who mammographic screening is targeted at or designed for (e.g., age requirements); details of the reporting process and timeline, and the fact that if further investigation is required, patients will be contacted. Anecdotally, the impression is that GPT4 provided a better written and more appropriate information sheet.

GPT3.5 generated plausible information for a screening mammography patient information sheet; however, the document contains errors, misinformation, and omits key information, making it inappropriate for education and health promotion. GPT4 produced a more appropriate information sheet, although omissions were still an issue. Reliance on ChatGPT by patients as a source of information, based on the use of search engines as a source of medical advice (e.g., “Dr Google”), is likely to become more common. It has been reported that before contacting a medical practitioner, 87% of Australian patients access medical information using search engines such as Google and WebMD. It has also been reported that seeking health information using the internet is significantly underpinned by trust in the site (Marcu et al., 2018). Whether this trust translates to ChatGPT is unknown, especially given concerns about the data sets used in its training. In addition, the ChatGPT model operates in a self-contained manner, lacking the capability to copy from other sources and access the internet for information retrieval when generating responses; instead, it predicts the next part of a sentence using patterns learned during training. The real-time generation of answers to questions in a convincing way obviates the need for trawling through multiple pages on a search engine report; but still requires a high degree of health literacy equivalent to that needed to review websites. As evidenced, the information provided by ChatGPT does not currently equate to accurate information or information that is understandable or actionable. This presents a challenge for using ChatGPT as an information source in health education settings where accuracy and clarity are critical. ChatGPT-generated content might not meet the standards for conveying accurate and understandable health-related information.

A recent study by Walker et al. (2023) reported that the reliability of medical information provided by ChatGPT answers compared to guideline recommendations across five medical conditions was 60%. Consequently, this lack of reliability could increase the demands on health care educators and staff, including radiologists and radiographers, to spend additional time clarifying misinformation and misconceptions produced by ChatGPT information. But a greater concern is that the poor understandability and actionability of the information provided by ChatGPT3.5 and ChatGPT4.0 combined with a low CDC Index score for actionability may result in women not participating in the screening program, putting them at risk of missed breast cancer and death (Table 2). A number of “ChatGPT type” errors or mistakes were detected in the GPT3.5 patient information sheet. This includes the use of outdated terminology (e.g., referring to “plates”). The errors included a variety of previously defined ChatGPT learning inaccuracies (Currie, 2023), including “hallucination” (incorrect information presented as fact by the chatbot), and content that is plausible but incorrect upon checking. Furthermore, there was an AI “illusion error of similarity” (e.g., the reporting process is similar but different), “delusion” as a result of AIs false beliefs producing wrong information (e.g., the implication that the equipment positions the patient rather than a radiographer), “delirium” with sophisticated answerers generated by AI from a confused algorithm (e.g., a small risk of radiation exposure during the procedure rather than a small risk from radiation during the procedure), “confabulation of fake information” (e.g., the fabricated Bleyer citation), and the extrapolation of a logical, although incorrect, extension of known information (e.g., advice regarding jewelry removal as an extrapolation from other imaging procedures).

These errors were reduced in the GPT4 version of the information sheet but it still included incorrect terminology (e.g., plate) and inaccurate protocol (e.g., two projections instead of four). GPT4 was more patient-facing in language and structure, less U.S.-centric in content, more succinct, and provided a disclaimer, advice, and medical information.

ChatGPT is known for the plausibility of the information it provides rather than its accuracy. The propensity to present incorrect information as fact occurs when the model has had to guess combinations of information as the detail of the original information is lost (hallucination) (Sobieszek & Price, 2022). In this study, ChatGPT has fabricated plausible-looking references that do not exist and several that support the information provided. ChatGPT cannot analyze the sources it uses for accuracy and, as such, is providing educational material without a source of truth.

The primary purpose of a patient information sheet in screening mammography is to educate women by providing sufficient information to make an informed decision about participation in screening mammography and undertaking the procedure. The BreastScreen NSW patient information sheets have been developed for understandability, actionability, and clear communication (Table 1) to attract women in the BreastScreen target age group to participate in screening taking into account their own perspectives (Woolcott Research, 2014).

The information sheet is also an important generator of patient questions, which is a crucial part of informed consent across all health literacies. Using ChatGPT for mammography information risks affecting patient involvement and experience due to inaccuracies. ChatGPT might be better suited to language conversion of exemplar clinical patient information sheets where English is a second language, although ChatGPT has been reported to be inferior in non-English language text generation (Seghier, 2023). Comparison across the multiple languages of the BreastScreen NSW exemplar was outside the scope of this study. Nonetheless, GPT4 provided an improved information sheet over GPT3.5, which could soon be deemed fit for purpose with continued training. However, reliance remains on the reader to assess the plausibility of the information provided, which may not suit all levels of health literacy.

A limitation of this study is that it did not include end users (the general public) in the PEMAT-P or CDC Index analysis, and the results may be biased due to the authors’ level of health literacy and clinical experience in this field.

Implications for Practice

ChatGPT has been shown to be an unreliable and inaccurate source of information concerning breast screening and may, as a consequence, not prompt the behavioral changes needed to reduce morbidity and mortality from breast cancer if used as a standalone source of truth. As a result, support for patients seeking additional information ahead of screening mammography is needed as it risks misinformation and confusion. This is of concern as it has implications for participation and practice. When women seek health education relating to mammography screening from ChatGPT, participation may decrease, and it is also foreseeable that there may be increased demands on the health care system to address ChatGPT’s inaccuracies. Also, health care educators may need to rectify information deficiencies. This is counter to the purported advantages of ChatGPT, and it could decrease efficiency.

Conclusion

ChatGPT is freely available and is an exciting and innovative tool for information generation and answering patient questions, but educators must be aware that it currently has limited accuracy when generating patient information for screening mammography, with the potential to negatively impact breast screening education and program participation. Access to ChatGPT4 is reported to have increased accuracy in the information provided, presenting a secondary barrier to health education for those who cannot afford it. Continued training of ChatGPT and the release of more advanced freely and publicly available versions of the AI chatbot may improve the PEMAT-P and CDC Index scores against benchmark health promotion material to a point where ChatGPT becomes a trusted easily accessible source of information for all health literacies. Health educators must recognize that Chat GPT is not at that point yet.