Sage Journals: Discover world-class research

Abstract

Five university-based research groups competed to recruit forecasters, elicit their predictions, and aggregate those predictions to assign the most accurate probabilities to events in a 2-year geopolitical forecasting tournament. Our group tested and found support for three psychological drivers of accuracy: training, teaming, and tracking. Probability training corrected cognitive biases, encouraged forecasters to use reference classes, and provided forecasters with heuristics, such as averaging when multiple estimates were available. Teaming allowed forecasters to share information and discuss the rationales behind their beliefs. Tracking placed the highest performers (top 2% from Year 1) in elite teams that worked together. Results showed that probability training, team collaboration, and tracking improved both calibration and resolution. Forecasting is often viewed as a statistical problem, but forecasts can be improved with behavioral interventions. Training, teaming, and tracking are psychological interventions that dramatically increased the accuracy of forecasts. Statistical algorithms (reported elsewhere) improved the accuracy of the aggregation. Putting both statistics and psychology to work produced the best forecasts 2 years in a row.

Keywords

prediction judgment decision making forecast social interaction

Get full access to this article

View all access options for this article.

References

Ariely

Tung Au

Bender

R. H.

Budescu

D. V.

Dietz

C. B.

. . . Zauberman

(2000). The effects of averaging subjective probability estimates between and within judges. Journal of Experimental Psychology: Applied, 6, 130–147.

Arkes

(1991). Costs and benefits of judgmental errors: Implications for debiasing. Psychological Bulletin, 110, 486–498.

Baron

Ungar

Stone

Mellers

Tetlock

(in press). Two reasons to make aggregated probability forecasts more extreme. Decision Analysis.

Benson

Onkal

(1992). The effects of feedback and training on the performance of probability forecasters. International Journal of Forecasting, 8, 559–573.

Betts

J. R.

Shkolnik

J. L.

(2000). Key difficulties in identifying the effects of ability grouping on student achievement. Economics of Education Review, 19, 21–26.

Bikhchandani

Hirshleifer

Welch

(1992). A theory of fads, fashion, custom, and cultural change as informational cascades. Journal of Political Economy, 100, 992–1026.

Brier

G. W.

(1950). Verification of forecasts expressed in terms of probability. Monthly Weather Review, 78, 1–3.

Epple

Romano

R. E.

(2011). Peer effects in education: A survey of the theory and evidence. In Benhabib

Bisin

Jackson

M. O.

(Eds.), Handbook of social economics (Vol. 1B, pp. 1053–1163). Amsterdam, The Netherlands: North-Holland.

Esser

J. K.

(1998). Alive and well after 25 years: A review of groupthink research. Organizational Behavior and Human Decision Processes, 73, 116–141.

10.

Fischhoff

Chauvin

(Eds.). (2011). Intelligence analysis: Behavioral and social scientific foundations. Washington, DC: National Academies Press.

11.

Fischhoff

Slovic

Lichtenstein

(1978). Fault trees: Sensitivity of estimated probabilities to problem representation. Journal of Experimental Psychology: Human Perception and Performance, 4, 330–344.

12.

Fong

Nisbett

Krantz

(1986). The effects of statistical training on thinking about everyday problems. Cognitive Psychology, 18, 253–292.

13.

Hertel

Kerr

N. L.

Messé

L. A.

(2000). Motivation gains in performance groups: Paradigmatic and theoretical developments on the Köhler effect. Journal of Personality and Social Psychology, 79, 580–601.

14.

Herzog

S. M.

Hertwig

(2009). The wisdom of many in one mind: Improving individual judgments with dialectical bootstrapping. Psychological Science, 20, 231–237.

15.

Internet Movie Script Database. (2011). Zero dark thirty. Retrieved from http://www.imsdb.com/scripts/Zero-Dark-Thirty.html

16.

Janis

(1982). Group-think. Boston, MA: Houghton-Mifflin.

17.

Johnson

Bruce

(2001). Calibration of subjective probability judgments in a naturalistic setting. Organizational Behavior and Human Decision Processes, 85, 265–290.

18.

Kahneman

Miller

(1986). Norm theory: Comparing reality to its alternatives. Psychological Review, 93, 136–153.

19.

Keren

(1987). Facing uncertainty in the game of bridge: A calibration study. Organizational Behavior and Human Decision Processes, 39, 98–111.

20.

Kerr

N. L.

MacCoun

R. J.

Kramer

G. P.

(1996). Bias in judgment: Comparing individuals and groups. Psychological Review, 103, 687–719.

21.

Kerr

N. L.

Tindale

R. S.

(2004). Group performance and decision making. Annual Review of Psychology, 55, 623–655.

22.

Koriat

Lichtenstein

Fischhoff

(1980). Reasons for confidence. Journal of Experimental Psychology: Human Learning and Memory, 6, 107–118.

23.

Larrick

R. P.

(2004). Debiasing. In Koehler

D. J.

Harvey

(Eds.), Blackwell handbook of judgment and decision making (pp. 316–337). Malden, MA: Blackwell.

24.

Latané

Williams

Harkins

(1979). Many hands make light the work: The causes and consequences of social loafing. Journal of Personality and Social Psychology, 37, 822–832.

25.

Levine

J. M.

Moreland

R. L.

(1990). Progress in small group research. Annual Review of Psychology, 41, 585–634.

26.

Lichtenstein

Fischhoff

(1980). Training for calibration. Organizational Behavior and Human Performance, 26, 149–171.

27.

Lovallo

Clarke

Camerer

(2012). Robust analogizing and the outside view: Two empirical tests of case-based decision making. Strategic Management Journal, 33, 496–512.

28.

McGrath

(1984). Groups: Interaction and performance. Englewood Cliffs, NJ: Prentice Hall.

29.

Mellers

B. A.

Stone

Metz

S. E.

Ungar

Tetlock

(2013). The psychology of intelligence analysis: Drivers of prediction accuracy in world politics. Manuscript submitted for publication.

30.

Murphy

A. H.

Winkler

R. L.

(1984). Probability forecasting in meteorology. Journal of the American Statistical Association, 79, 489–500.

31.

Murphy

A. H.

Winkler

R. L.

(1987). A general framework for forecast verification. Monthly Weather Review, 115, 1330–1338.

32.

O’Hagan

Buck

C. E.

Daneshkhah

Eiser

J. R.

Garthwaite

P. H.

Jenkinson

D. J.

. . . Rakow

(2006). Uncertain judgements: Eliciting experts’ probabilities. New York, NY: Wiley.

33.

Page

(2007). The difference: How the power of diversity creates better groups, firms, schools, and societies. Princeton, NJ: Princeton University Press.

34.

Satopää

V. A.

Baron

Foster

D. P.

Mellers

B. A.

Tetlock

P. E.

Ungar

L. H.

(2014). Combining multiple probability predictions using a simple logit model. International Journal of Forecasting, 30, 344–356.

35.

Satopää

V. A.

Jensen

S. T.

Mellers

B. A.

Tetlock

Ungar

(in press). Aggregation in time-series: Dynamic hierarchical modeling of sparse expert beliefs. Annals of Applied Statistics.

36.

Soll

J. B.

Larrick

R. P.

(2009). Strategies for revising judgment: How (and how well) people use others’ opinions. Journal of Experimental Psychology: Learning, Memory, and Cognition, 35, 780–805.

37.

Soll

J. B.

Milkman

K. L.

Payne

J. W.

(in press). A user’s guide to debiasing. In Wu

Keren

(Eds.), Handbook of judgment and decision making. New York, NY: Wiley.

38.

Stasser

Titus

(1985). Pooling of unshared information in group decision making: Biased information sampling during discussion. Journal of Personality and Social Psychology, 48, 1467–1478.

39.

Sunstein

(2006). Infotopia: How many minds produce knowledge. Oxford, England: Oxford University Press.

40.

Tetlock

P. E.

Mellers

B. A.

(2011). Intelligent management of intelligence agencies: Beyond accountability ping-pong. American Psychologist, 66, 542–554.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.57 MB

0.82 MB

Psychological Strategies for Winning a Geopolitical Forecasting Tournament

Abstract

Keywords

Get full access to this article

References

Supplementary Material