Forewarning Artificial Intelligence about Cognitive Biases

Abstract

Artificial intelligence models display human-like cognitive biases when generating medical recommendations. We tested whether an explicit forewarning, “Please keep in mind cognitive biases and other pitfalls of reasoning,” might mitigate biases in OpenAI’s generative pretrained transformer large language model. We used 10 clinically nuanced cases to test specific biases with and without a forewarning. Responses from the forewarning group were 50% longer and discussed cognitive biases more than 100 times more frequently compared with responses from the control group. Despite these differences, the forewarning decreased overall bias by only 6.9%, and no bias was extinguished completely. These findings highlight the need for clinician vigilance when interpreting generated responses that might appear seemingly thoughtful and deliberate.

Highlights

• Artificial intelligence models can be warned to avoid racial and gender bias.

• Forewarning artificial intelligence models to avoid cognitive biases does not adequately mitigate multiple pitfalls of reasoning.

• Critical reasoning remains an important clinical skill for practicing physicians.

Human-like cognitive biases may cause artificial intelligence large language models (LLMs) to produce faulty medical recommendations. ¹ For example, model recommendations tend to be more risk averse when treatment outcomes are described by mortality statistics rather than survival statistics (framing effects). These biases can be difficult to correct because they are frequent in published text, inherent to common algorithms, and reinforced by human calibration.

Regular patients may consult LLMs for medical problems and subsequently act on mistakes generated by artificial intelligence. Initial studies suggest formulating a request more carefully with an early explicit warning might mitigate such biases in non-medical domains (term: prompt engineering). ² In this study, we tested whether a forewarning might reduce biases from an LLM generating clinical recommendations for medical care (Appendix §1).

Methods

We presented 10 clinical cases to OpenAI’s LLM GPT-4 (model gpt-4-0613). ³ Each case (Appendix §6) appeared in 2 versions that differed by introducing a specific cognitive bias, such as framing effects (e.g., surgery outcomes described with survival rather than mortality statistics). For the intervention group, the case also contained the forewarning, “Please keep in mind cognitive biases and other pitfalls of reasoning.” For the control group, the case contained no forewarning (Figure 1). In all other ways, the LLM settings were identical with mid-range randomness, no learning examples, zero conversation history, and only text-based information.

Figure 1

Example of forewarning for cognitive bias.

Each case version was presented to synthetic physicians (N = 100) simulated by GPT-4’s system prompt, “You are a physician.” No further physician characteristics were assigned because past research suggested these characteristics have minimal effect on cognitive biases. ¹ Synthetic physicians generated clinical recommendations, which were categorized according to a priori criteria (Appendix §2, §3). We then analyzed responses by comparing the intervention and control groups with the t test, chi-square, and 2-proportion z test as appropriate to quantify the extent of bias. The study protocol received a waiver from the Sunnybrook Research Ethics Board.

Results

Synthetic physician responses after a forewarning, compared with controls, were more than 50% longer (213 words v. 138 words, P < 0.001) and discussed a cognitive bias more than 100 times more frequently (44.3% v. 0.4%, P < 0.001). A bias forewarning, however, identified the correct bias in only 5% of responses (Appendix §3). The case assessing emergency department care had the most relevant discussion, wherein 46 of 200 responses (23%) correctly identified hindsight bias. The most frequently discussed bias after forewarning was anchoring bias, which appeared in 320 of 2,000 responses (16%) yet was irrelevant to all 10 clinical cases.

In the lung cancer case, synthetic physicians who received a forewarning subsequently identified framing effects correctly in 4 of 200 responses (2%) and discussed an irrelevant bias in 33 of 200 responses (17%). One of the few relevant responses stated, “It’s also worth noticing potential cognitive biases here, such as the framing effect (whether survival rates or mortality rates are emphasized) . . . .” In contrast, one irrelevant response stated, “It is important to be aware of anchoring bias here, which is an over-reliance on initial pieces of information.” In 163 of 200 responses (79%), no bias was discussed despite the forewarning.

On average, forewarning reduced cognitive biases by 6.9% (Figure 2). A modest example was framing effects, for which the bias was reduced by about one-third after a forewarning (66% v. 45%, P = 0.003). The strongest example was the aggregate-individual discrepancy, for which the bias was reduced by nearly half after forewarning (52% v. 27%, P < 0.001). In contrast, forewarning increased Occam’s razor fallacy (77% v. 89%, P = 0.024). Notably, synthetic physicians were immune to base-rate neglect and capitulating-to-pressure regardless of forewarning. No bias present in the control group was eliminated after a forewarning (Appendix §4, §5).

Figure 2

Forewarning effect on cognitive bias.

Discussion

We found only minor improvements in cognitive biases after testing an explicit forewarning, despite generating lengthier and seemingly thoughtful responses. Furthermore, the LLM struggled to identify the correct cognitive bias, defaulting instead to the anchoring effect, availability bias, or the placebo effect (perhaps due to high frequency in training corpora). Biases persisted after forewarning and varied in extent for a range of medical judgments that included differential diagnoses, preoperative assessment, and disposition planning. Ultimately, LLMs may lull patients to complacency by exhibiting increased deliberation and discussing irrelevant biases. ⁴

Cognitive biases can arise from several sources inherent to LLMs. Training corpora can contribute if common narratives are not fully accurate (e.g., framing effects due to text emphasizing successful operations and “survivor” sentiments). ⁵ Algorithms may prioritize information according to attention mechanisms favored by fallible individuals (e.g., primacy effects due to topic sentences building context). ⁶ Human calibration might also guide models with variable success (e.g., applying Bayesian equations to eliminate base-rate neglect). These factors contribute to the magnitude of cognitive bias and highlight opportunities for future research.

Our study has limitations because we tested a single concise bias forewarning yet other alternative strategies might have potential. We tested nuanced open-ended clinical decisions, yet a forewarning may potentially be more effective for esoteric topics with closed-ended correct answers. ⁷ We tested a model trained on a broad proprietary corpus, yet it might not fully capture the diversity of practicing physicians. ⁸ We tested 1 model, yet other LLMs may have more safeguards. Despite these limitations, our study suggests the cognitive biases of an LLM have no simple remedy and thereby underscores the role of active clinicians with critical reasoning skills.

Supplemental Material

sj-docx-1-mdm-10.1177_0272989X251346788 – Supplemental material for Forewarning Artificial Intelligence about Cognitive Biases

Supplemental material, sj-docx-1-mdm-10.1177_0272989X251346788 for Forewarning Artificial Intelligence about Cognitive Biases by Jonathan Wang and Donald A. Redelmeier in Medical Decision Making