Coherence and Correspondence Competence: Implications for Elicitation and Aggregation of Probabilistic Forecasts of World Events

Abstract

One potentially useful concept that arises in the elicitation and aggregation of probabilistic forecasts is Hammond’s (1996) distinction between coherence and correspondence. A study was conducted to test the commonly held assumption that coherence competency, a judge’s ability to reason correctly according to the prescriptions demanded by the problem, directly yields correspondence competency, a judge’s ability to predict the outcome that actually happens in the external world. The role of a visualization aid in terms of moderating these effects was also examined. Participants who were knowledgeable baseball fans predicted the probability with which their favored team would win the 2011 Major League Baseball World Series, giving a prior probability shortly before the start of the Series, and then sequentially updating their answer as the individual games unfolded over time. Results show that for participants using the visualization, their ability to update probabilities according to the dictates of Bayes’ Theorem was correlated with their ability to predict the winner of the 2011 MLB Series – a desirable property that allows for estimation of judges’ outcome performance based on more readily available process information.

Get full access to this article

View all access options for this article.

References

Brier

G. W.

(1950). Verification of forecasts expressed in terms of probability. Monthly Weather Review, 78, 1-3.

Ericsson

K. A.

Krampe

R. T.

Tesch-Romer

(1993). The role of deliberate practice in the acquisition of expert performance. Psychological Review, 100, 363-406.

Gigerenzer

Hoffrage

(1995). How to improve Bayesian reasoning without instruction: Frequency formats. Psychological Review, 102, 684-704.

Hammond

K. R.

(1996). Human judgment and social policy: Irreducible uncertainty, inevitable error, unavailable injustice. New York: Oxford University Press.

Surowiecki

(2004). The wisdom of crowds. New York: Doubleday.

Tetlock

P.E.

(2005). Expert political judgment: How good is it? How can we know? Princeton: Princeton University Press.

Tsai

Kirlik

Miller

Kosorukoff

(2008). Rule and instance based strategies in expert judgment. Proceedings of the Human Factors and Ergonomics Society 52nd Annual Meeting. New York, NY.

Tsai

Miller

Kirlik

(2011). Interactive visualizations to improve Bayesian reasoning. Proceedings of the Human Factors and Ergonomics Society 55th Annual Meeting. Las Vegas, NV.

Tversky

Kahneman

(1974). Judgment under uncertainty: Heuristics and biases. Science, 185, 112-1131.