Decisive Evidence on a Smaller-Than-You-Think Phenomenon

Abstract

Accurate perception of medical probabilities communicated to patients is a cornerstone of informed decision making. People, however, are prone to biases in probability perception. Recently, Pighin and others extended the list of such biases with evidence that “1-in-X” ratios (e.g., “1 in 12”) led to greater perceived probability and worry about health outcomes than “N-in-X*N” ratios (e.g., “10 in 120”). Subsequently, the recommendation was to avoid using “1-in-X” ratios when communicating probabilistic information to patients. To warrant such a recommendation, we conducted 5 well-powered replications and synthesized the available data. We found that 3 out of the 5 replications yielded statistically nonsignificant findings. In addition, our results showed that the “1-in-X” effect was not moderated by numeracy, cognitive reflection, age, or gender. To quantify the evidence for the effect, we conducted a Bayes factor meta-analysis and a traditional meta-analysis of our 5 studies and those of Pighin and others (11 comparisons, N = 1131). The meta-analytical Bayes factor, which allowed assessment of the evidence for the null hypothesis, was very low, providing decisive evidence to support the existence of the “1-in-X” effect. The traditional meta-analysis showed that the overall effect was significant (Hedges’ g = 0.42, 95% CI 0.29–0.54). Overall, we provide decisive evidence for the existence of the “1-in-X” effect but suggest that it is smaller than previously estimated. Theoretical and practical implications are discussed.