Teaching Medical Students to Communicate Risks Like Military Intelligence Analysts

Abstract

Communication about health often involves descriptions of risk: the probability or likelihood of an unfavorable outcome. Communicating risk helps individuals make choices about their own health by building understanding of potential outcomes and providing context for the importance of procedures, health interventions, and lifestyle choices. However, medical education in the United States does not provide future physicians with adequate statistical literacy to communicate risk effectively and rarely encourages them to practice communicating risk in pre-clinical years. Risk communication in military intelligence, a field with formalized risk language and training, offers a unique perspective into potential improvements for medical risk communication. With backgrounds in the military, public health, communication, surgery, and medical education, the authors offer the following recommendations to improve risk communication for medical students. (1) Encourage the use of numerical absolute risk when communicating among health practitioners to avoid varied interpretations of what different risk descriptors (“uncommon,” “likely,” or “low”) might mean; (2) build efficient, teachable skills in use of patient-facing risk communication tools like comparative probabilities and visual aids; and (3) practice estimating risk through role-play of risk communication between medical students and standardized patients. By improving risk communication in medical education, future doctors will be better equipped to build trust through open communication and improve the health of the patients and the communities for whom they care.

Introduction

In the early 2000s, the Central Intelligence Agency (CIA) published what would become the key assessment used to justify the invasion of Iraq. ¹ These assessments are now well known to be incorrect. One of the major issues with the CIA's document is that its executive summary removed the probability qualifiers that existed in the body of the report, thereby communicating an unjustified level of certainty. ² Only on the final page of the executive summary do any qualifiers exist, but these still fail to provide an absolute risk corollary, leaving their exact probability up to the reader's assessment and bias.^1,2

Risk communication is critical to health

The CIA's assessment failure that led to the invasion of Iraq demonstrates the importance of communicating risk and probability clearly and understandably. As in military intelligence, risk communication is an essential part of clinical medicine and public health. For patients to engage in decision-making and informed consent, they must understand potential outcomes and their associated likelihoods with acknowledged uncertainty.^3,4 In medicine, risk is described as the probability or likelihood of a bad outcome that is associated with action or inaction, such as whether to choose a treatment, procedure, or health intervention or maintain the status quo. ⁵

Challenges of risk literacy

Risk literacy in medicine is the skill necessary to weigh potential outcomes related to specific treatment or intervention decisions. ⁶ In public health and clinical medicine, it is challenging to attain and maintain risk literacy without a requisite knowledge of probability and statistics.

Physicians are expected to understand medical risk information as well as the added complexity of P-values and confidence intervals and explain them in lay terms to patients within the limited timeframe of a patient interaction. ⁷

Gaps in education around risk

Studies consistently find that medical students and residents lack basic statistical skills.^8–17 These deficiencies persist when students graduate into clinical practice; across various disciplines, established physicians still demonstrate deficient understanding of statistical concepts and overestimate harms and benefits of treatments at a similar rate to residents or medical students.^12–15

As expected, lower physician risk literacy results in misleading risk communication with patients.^15,16 Newman et al found only one-third of physician-patient pairs (n = 404) held similar (within 10%) estimates of absolute risk for the patient, notably after discussing the patient's case and treatment together. ¹⁸

Addressing poor risk literacy among physicians should start in medical school. Medical education in the United States does include concepts related to risk, but much of it occurs as a secondary objective in other lectures rather than focused lectures or in role-playing patient scenarios. ¹⁹ Medical students, however, do not need much focused time to improve in various metrics of risk literacy: the median score on the Quick Risk Test increased from 50% to 90% among 169 medical students after just one 90-min lecture.^10,11 Positive framing of risk lectures as “communication trainings” rather than “statistics seminars” has also been shown to increase scores significantly on the Quick Risk Test. ²⁰

Building from lessons in military intelligence, research findings, and educational interventions, we argue that improved risk communication is attainable in medical students by standardizing communication and teaching and testing best practices for patient-facing risk communication with efficient practical application.

How to teach risk communication in medical education

In 2011, President Obama began receiving intelligence estimates about the location of Osama bin Laden. The US intelligence community was still reeling from its failure in Iraq but had learned to present likelihood in terms of absolute risk.

In the spring of 2011, President Obama received 4 assessments from different intelligence agencies on the likelihood that the compound in question in Pakistan housed bin Laden: 40%, 60%, 80%, and 95%. ²¹ The president expressed his frustration that these assessments were different and determined he would have to make his own estimate of likelihood. President Obama thought the odds were “50/50” and authorized the raid that would eventually kill Osama bin Laden.

However, without weighing individual probabilities differently, a set of probabilities is equivalent to their simple average. President Obama should have understood the probability from his 4 agencies to be 69%, substantially higher than his own assessment of 50/50. ²¹

Non-standard risk presentation increases confusion

President Obama's frustration with his intelligence agencies is analogous to patients’ annoyance with doctors who are unable to give clear, understandable answers to questions on risk or treatment outcomes. The variety of language and methods used to describe even the same absolute risk percentages causes confusion, frustration, and potential mistrust between patients and their providers.^22,23 Should a 1% absolute risk be described as negligible, very low, low, minimal, uncommon, or rare? In different contexts, 1% absolute risk might also be considered acceptable or unacceptable. ²⁴ A drug with a 1% absolute risk of debilitating side effects might be acceptable if it can relieve debilitating pain, but not acceptable if the drug treated asymptomatic hypertension. ²⁵ The time course of when a bad outcome might occur can also affect an individual's perception of acceptable risk. A physician describing absolute risk percentages to a patient must consider context, time course, and overall uncertainty, which allows for wide variation in risk presentation. These variations are known to affect decision-making for both health practitioners and patients. ²⁶ In cases where patients have complications from a procedure or develop side effects from a drug, the clarity of the risk discussion they had with their physician could determine whether they feel misled, grow distrustful of medicine, or take legal action against the treating institution.

Standardizing risk language has been offered as a possible solution to the complexities of risk communication. In 1996, Sir Kenneth Calman, then England's Chief Medical Officer, first proposed a standard classification of risk language, arguing that such classification allows for improved personal risk-benefit analysis since individuals, depending on context, will accept varying levels of risk for different perceived benefits. ²⁴ In Calman's words, “negligible” could only be used for absolute risks <1 in a million (<0.0001%), “moderate” could only be used for absolute risks from 1 in 1000 to 1 in 100 (0.1%-1%), with “minimal,” “very low,” and “low” in between. ²⁴

Standardizing risk and probability language has a related history in military intelligence. Years before Calman and the CIA's failure in Iraq, Sherman Kent, then a professor of history at Yale and an accomplished US CIA analyst, made a similar request to standardize probability and risk language in US intelligence analysis. ²⁷ During the Cold War, Kent argued that variations in language (such as “serious possibility”) resulted in significant confusion with life-and-death consequences due to the mis-estimation of Soviet intentions to invade its neighbors and the resulting delay in the United States helping fortify their defenses. Kent's suggestion was codified in 2007 by the Director of National Intelligence into an Intelligence Community Directive, which standardized the language used in intelligence writing and presentation. ²⁸ Today, intelligence analysts are trained to these standards and tested to ensure consistent understanding of what their terminology means in absolute risk terms.

Standardizing medical risk communication

In contrast, Calman's proposal for standardization in medicine has had limited uptake. A study in 1998 by Edwards et al found that primary care professionals thought standardization of risk language might be helpful for provider-to-provider communication but was too rigid for communicating with individual patients given their differing communication needs and styles. ²⁹ In the early 2000s, the European Union (EU) formalized the absolute risk ranges associated with words like “common” and “rare” for print labels of medication side effects; however, only 7 of 180 participants from the general public estimated the correct absolute risk percentages to these risk terms. ³⁰ A follow-up study comparing the EU and Calman guidelines found that both doctors and patients significantly overestimated risk when given non-numerical risk descriptions. ³¹

Why Kent's proposal was successful and Calman's was not may have to do with the different contexts and uses of risk language in medicine versus military intelligence. Calman's context was cancer, where 1% lifetime risk of developing a cancer could be considered a “moderate” or “high” risk. Contrast that with a 1% risk of complications during a lifesaving surgery, which might be considered “low” or “minimal” risk. Within other fields of medicine, different scoring systems also exist for diagnostic assessment, such as Wells’ Criteria for pulmonary embolism, which uses terms like “unlikely” (less than 8% chance of having a pulmonary embolism) and “likely” (greater than 8%), and the pre-test probability of obstructive coronary artery disease which defines risk as “low,” “intermediate,” and “high” with <15%, 16% to 50%, and >50%, as their respective qualifications of absolute risk. ^{32 ,33} Standardizing these scoring languages would detract from their primary purpose in immediate clinical judgment and intervention. Figure 1 displays the differences between the absolute risk and these various scoring systems, compared against both the standard derived from Kent's work (ICD 203) and understandable common comparative probabilities.

Figure 1.

The different words used to describe the same absolute risk percentage (far left column) based on different contexts: from left to right, Calman in cancer, Wells in pulmonary embolism, CAD for coronary artery disease, and ICD 203 military intelligence.^24,28,32,33 common comparative probabilities are included as examples of how to make more abstract numbers, like 0.1% or even 5%, gain relevance to everyday life.^34–36

Military intelligence training emphasizes the importance of communicating absolute risk in understandable terms. The first lesson from the CIA's failure in Iraq is that absolute risk should be included in all assessments to avoid miscommunication or unjustified certainty, regardless of the use of standardized probability language. Even when absolute risk is applied, military intelligence analysts are taught to communicate 1 probability rather than a range or set of different assessments, as was learned from the case presented to President Obama.

Medical education should apply these same lessons to risk communication for medical students. Absolute risk is more effective than various descriptions of probability, and a range of absolute risk assessments is less informative than their simple average if no patient-specific information tilts the range one way or the other. ³⁷ Absolute risk has been shown to increase both patient and provider understanding of outcomes, especially compared to relative risk.^9,23,38

This practice should begin in medical school. Students can gain a deeper understanding of the risk of complications by developing a knowledge of the associated absolute risks rather than more cursory statements like “side effects are rare.”

Teaching risk communication best practices

While consistent use of absolute risk in communication with other providers can be enforced and tested during the clinical years of medical education, the pre-clerkship curriculum should include best practices and tools for patient-facing communication, such as comparative probabilities and visual aids, and evaluate their use through practical application. When offered in addition to numerical risk, narratives can enhance communication, and metaphors provide understandable and meaningful depictions of risk when used correctly.^3,39 Visual aids (Figure 2) can make narratives describing risk both clearer and more accurate, as they provide direct depictions of the described risk in graphical form. ³ Pairing qualitative and quantitative descriptions of risk together also communicates risk most effectively, as shown in a large review of prescription drug labeling. ³⁷ Using visual aids and comparative probabilities as teaching aids in curricula on evidence-based medicine or critical appraisal skills could demonstrate to students how to use them in communicating with patients.^11,40

Figure 2.

Two visual aids that communicate risk information: an icon array on the left and a bar chart on the right. The icon array on the left depicts a commonly used example of screening mammography, adapted from data in Germany by Hoffrage and Koller to show realistic benefits and limitations of screening. ⁴¹ The circles represent 200 women who are 50–69 years old, and in this population without screening, 4/200 will die from breast cancer (colored circles). If all women underwent screening, 1 fewer woman would die (gray circle). The icon array better explains the risk information than the notable variation between relative risk reduction (25%), absolute risk reduction (0.5%), and number needed to treat/screen (200). A doctor encouraging mammograms despite this information could say mammograms “provide a small likelihood of a big impact” as the impact would be significant for the 1 woman saved from breast cancer by screening. The bar graph on the right shows percentage estimates given by physicians and patients of mortality (black) at home from myocardial infarction, adapted from data in Newman et al. ¹⁸ Newman et al showed patients significantly overestimated the risk of mortality at home (80%) compared to physicians (15%), even after the risks were discussed together with their physician. Using a bar graph as above to communicate the physician's risk estimate could alleviate this confusion.

Comparative probabilities—using frequencies of non-medical events known in everyday life (Figure 1)—allow physicians to make risk more tangible for patients. For example, a physician could compare rare (0.1%-0.01%) side effects with the odds of guessing a 3-digit 2-factor authentication code or pitching a “perfect” game in baseball. Prerequisites for this type of comparison include a familiarity with common event frequencies, which could be provided as part of the curriculum on risk communication, and a consistent understanding of absolute risk not biased by experience or anecdotal evidence. Including the numerical estimate of the patient's risk in the discussion would help ensure the chosen comparison is not misleading. ⁴²

Visual aids (Figure 2) serve as transparent depictions of probability. ⁴⁰ Icon arrays, sets of shapes either empty or filled in to show frequency, were shown to close the gap in terms of understanding and communicating risk between surgeons with high- and low-risk literacy. ¹³ Colored bar charts were also found effective by general practitioners and patients for communicating absolute risk before and after treatment in cardiovascular disease. ⁴³ Physicians can further pair these graphical portrayals with size references. To describe a 1 in 200 risk, a health practitioner could compare it to the size of the patient's high school class. A size reference could also be a family of 5 for a more common risk or an entire football stadium of spectators for something much less likely.

Practice estimating risk

Practice estimating likelihood through source reliability, information credibility, and intelligence formulas is common in military intelligence training. Two examples from wars in the twenty-first century demonstrate how critical this practice is.

In the War Against the Islamic State in Iraq and Syria (ISIS), intelligence analysts from the United States and its allies guarded the Iraqi frontline from ISIS with aerial surveillance via drones. ⁴⁴ In 2017, a drone operator observed several suspected ISIS fighters flying a quadcopter around a known ISIS hideout. Intelligence analysts presented 2 possible explanations: a normal training flight for an ISIS operator or a test flight prior to launching an attack on the Iraqi frontline. The intelligence analysts gave a 50/50 assessment initially, admitting they did not have enough evidence to tilt 1 way or the other. The drone operator noted the fighters packing up a pick-up truck a few minutes later. Intelligence analysts shifted their assessment to 60/40 in favor of a test flight before an imminent attack. The drone operator then followed the pick-up truck as it drove away from the ISIS hideout. At an intersection where a left turn led toward the Iraqi frontline and a right turn deeper into ISIS territory, the truck turned left. Intelligence analysts gave the assessment of 95% likelihood of an attack on the Iraqi frontline. US commanders were notified, the Iraqis were warned, and the impending ISIS attack was successfully thwarted. ⁴⁴

The Russian military was conducting a military exercise on the borders of Ukraine while the world watched the Winter Olympics in Beijing in February 2022. Intelligence analysts immediately began assessing the possibility of an invasion. Intelligence collection focused on intercepted communications and aerial analysis of the “military exercise” to determine the likelihood of an invasion. US intelligence then discovered that Russia had moved significant blood and medical supplies near the border with Ukraine. ⁴⁵ This moved intelligence assessments closer to certainty that Russia was about to invade Ukraine, which occurred 4 days after the closing ceremonies in Beijing.

As these examples show, intelligence analysts continually gather data, assess its credibility via source and method of collection, and build and refine their assessments. Intelligence formulas can help balance or shift assessments with competing evidence for and against a certain outcome, as in the cases above where the left turn and blood supplies would have been weighed more heavily to tilt the assessment. This assessment process is notably similar for health practitioners, who must collect signs and symptoms from patients and estimate risk based on those symptoms and their reliability, as well as patient history and demographics. Like intelligence formulas, health practitioners have medical calculators like Wells’ Criteria at their disposal to weigh competing evidence.

In medical school, informal questions about relevant absolute risks, at the bedside, during rounds, or in the operating room, should become part of a longitudinal curriculum on evidence-based medicine or critical appraisal skills to help medical students build their skills in estimation. This informal questioning is a simple practice more senior physicians could include without significantly impacting the limited time available in the curriculum. Curricula also should encourage active participation in research, which has been shown to help students maintain statistical literacy and avoid bias from anecdotal evidence when making estimations. ⁴⁶

As military intelligence analysts practice their skills in risk estimations on mock enemy situations prior to analyzing true threats, the best way to practice risk communication in medicine is in practical application with standardized patients. Increased use of patient scenarios where a “patient” or “family member” asks for more clarification on the risk of a treatment or procedure would compel medical students to answer realistic and probing questions and deepen their understanding of how to communicate risk. ⁴⁷

Role-playing risk communication can be built into already existing curricula on evidence-based medicine or critical appraisal skills. Standardized patient encounters can be enhanced by adding grading items to describe whether the student communicated risk correctly and effectively. During lectures on risk, short breakout sessions where students practice with each other, switching roles between doctor and patient, could also reinforce the risk communication skills necessary to mitigate health misinformation at the individual level. ⁴⁸ Figure 3 provides some simple statements and questions a standardized patient could pose to a medical student with their associated learning objectives.

Figure 3.

Some basic lines on the left that a role-playing patient or family member can use in any health and treatment context to assess and evaluate a medical student's ability to communicate risk. The learning objectives on the right show some of the important end states medical students can reach through training in risk communication.

Conclusion

Lessons from military intelligence can improve medical education on risk communication and allow future doctors to help patients navigate the growing challenges of the complex digital health environment. Using absolute risk to communicate with other health practitioners will increase understanding of true frequency of outcomes and allow for improved risk communication. Training medical students to use comparative probabilities and visual aids will ensure they can provide patients with proper risk perceptions and be less dependent on their own or their patients’ risk literacy. Estimating risk, even while acknowledging uncertainty, is a skill that requires practice in medical school and refinement in the clinical years. Finally, role-playing these skills in risk communication with practical application represents the most impactful and time-efficient manner to train and improve student communication skills prior to seeing patients.

Medical education is already filled with lessons meant to improve health communication and overall patient outcomes. Our suggestions on risk communication are not time-intensive and can be incorporated into required curricula and established standardized patient encounters. Senior physicians also should use their time with medical students in clinical environments to quiz them on the absolute risk of outcomes for the diseases they are treating or procedures they are observing.

Like intelligence analysts, doctors must attempt to predict the future for their patients with imperfect reference data and variable uncertainty. Luckily, medical students can hone their skills in risk estimation and communication through verified, practical, and efficient training. With improved risk communication, future doctors will gain their patients’ trust and be more effective at improving their patients’ health.