Forecasting tournaments are level-playing-field competitions that reveal which individuals, teams, or algorithms generate more accurate probability estimates on which topics. This article describes a massive geopolitical tournament that tested clashing views on the feasibility of improving judgmental accuracy and on the best methods of doing so. The tournament’s winner, the Good Judgment Project, outperformed the simple average of the crowd by (a) designing new forms of cognitive-debiasing training, (b) incentivizing rigorous thinking in teams and prediction markets, (c) skimming top talent into elite collaborative teams of “super forecasters,” and (d) fine-tuning aggregation algorithms for distilling greater wisdom from crowds. Tournaments have the potential to open closed minds and increase assertion-to-evidence ratios in polarized scientific and policy debates.
GardnerD. (2011). (See References). Offers compelling contemporary and historical examples illustrating how it is possible for charlatans to flourish when we fail to systematically monitor judgmental accuracy.
2.
KahnemanD. (2011). (See References). Provides an elegant overview of the heuristics-and-biases research program, written by arguably the most influential psychologist of the last century.
3.
TetlockP. E. (2005). (See References). Reports the first large-scale effort to use psychological theories and methods to study judgmental accuracy in world politics.
4.
TetlockP. E. (2009, September/October). Reading tarot on K Street. The National Interest, 57–67. Makes the case for keeping objective score on political pundits and consultants promising to improve the quality of judgment.
5.
TetlockP. E.MellersB. A. (2011). (See References). Explores how difficult it is for people and organizations to commit to improving judgmental accuracy when they are caught in the middle of accountability blame games.
6.
ArmstrongJ. S. (Ed.). (2001). Principles of forecasting: A handbook for researchers and practitioners. New York, NY: Springer.
7.
AtanasovP.RescoberP.StoneE.SwiftS.Servan-SchreiberE.TetlockP.UngarL.MellersB. (2014). Crowd forecasting with prediction polls versus prediction markets. Manuscript in preparation.
8.
AxelrodR. M. (2006). The evolution of cooperation. New York, NY: Basic Books.
9.
BaronJ. (2000). Thinking and deciding. New York, NY: Cambridge University Press.
10.
BaronJ.UngarL. H.MellersB. A.TetlockP. E. (2014). Two reasons to make aggregated probability forecasts more extreme. Decision Analysis. Advance online publication. doi: 10.1287/deca.2014.0293
11.
BettsJ. R.ShkolnikJ. L. (2000). Key difficulties in identifying the effects of ability grouping on student achievement. Economics of Education Review, 19, 21–26.
12.
BielbyW. T. (2000). Minimizing workplace gender and racial bias. Contemporary Sociology, 29, 120–129.
13.
BielbyW. T. (2008). Promoting racial diversity at work: Challenges and solutions. In BriefA. P. (Ed.), Diversity at work (pp. 53–86). Cambridge, England: Cambridge University Press.
14.
BrierG. W. (1950). Verification of forecasts expressed in terms of probability. Monthly Weather Review, 78, 1–3.
15.
BudescuD.ChenE. (in press). Identifying expertise to extract the wisdom of crowds. Management Science. Advanced online publication. doi: http://dx.doi.org/10.1287/mnsc.2014.1909
16.
CronbachL. J. (1986). Social inquiry by and for earthlings. In FiskeD. W.ShwederR. A. (Eds.), Metatheory in social science: Pluralisms and subjectivities (pp. 83–107). Chicago, IL: University of Chicago Press.
17.
DawesR. M. (1998). Behavioral decision making and judgment. In GilbertD. T.FiskeS. T.LindzeyG. (Eds.), The handbook of social psychology (Vols. 1–2, 4th ed., pp. 497–548), New York, NY: McGraw-Hill.
18.
ErevI.ErtE.RothA. E.HaruvyE.HerzogS. M.HauR.LebiereC. (2010). A choice prediction competition: Choices from experience and from description. Journal of Behavioral Decision Making, 23, 15–47.
19.
GardnerD. (2011). Future babble: Why expert predictions are next to worthless, and you can do better. New York, NY: Dutton.
20.
GreenwaldA. G.PoehlmanT. A.UhlmannE. L.BanajiM. R. (2009). Understanding and using the Implicit Association Test: III. Meta-analysis of predictive validity. Journal of Personality and Social Psychology, 97, 17–41.
21.
HansonR. (2007). Logarithmic market scoring rules for modular combinatorial information aggregation. Journal of Prediction Markets, 1, 3–15.
22.
HartzmarkM. L. (1991). Luck versus forecast ability: Determinants of trader performance in futures markets. The Journal of Business, 64, 49–74.
23.
HunterJ. E.HunterR. F. (1984). Validity and utility of alternative predictors of job performance. Psychological Bulletin, 96, 72–98.
24.
JervisR. (1976). Perception and misperception in international politics (Vol. 49). Princeton, NJ: Princeton University Press.
25.
JervisR. (2010). Why intelligence fails: Lessons from the Iranian Revolution and the Iraq War. Ithaca, NY: Cornell University Press.
26.
KahnemanD. (2011). Thinking, fast and slow. London, England: Macmillan.
27.
KahnemanD.KleinG. (2009). Conditions for intuitive expertise: A failure to disagree. American Psychologist, 64, 515–526.
28.
KarvetskiC. W.OlsonK. C.MandelD. R.TwardyC. R. (2013). Probabilistic coherence weighting for optimizing expert forecasts. Decision Analysis, 10, 305–326.
29.
LernerJ. S.TetlockP. E. (1999). Accounting for the effects of accountability. Psychological Bulletin, 125, 255–275.
30.
LevittB.MarchJ. G. (1988). Organizational learning. Annual Review of Sociology, 14, 319–340.
31.
MacCounR. J. (1998). Biases in the interpretation and use of research results. Annual Review of Psychology, 49, 259–287.
32.
MakridakisS.AndersenA.CarboneR.FildesR.HibonM.LewandowskiR.WinklerR. (1982). The accuracy of extrapolation (time series) methods: Results of a forecasting competition. Journal of Forecasting, 1, 111–153.
33.
MellersB.HertwigR.KahnemanD. (2001). Do frequency representations eliminate conjunction effects? An exercise in adversarial collaboration. Psychological Science, 12, 269–275.
34.
MellersB.StoneE.AtanasovP.RohrbaughN.MetzS. E.UngarL.. . . TetlockP. (2014). The psychology of intelligence analysis: Drivers of prediction accuracy in world politics. Manuscript submitted for publication.
MooreD. A.SwiftS. A.MinsterA.MellersB. A.UngarL.TetlockP. E.. . . TenneyE. R. (2014). Confidence and accuracy over time. Manuscript in preparation.
37.
Open Science Collaboration. (2012). An open, large-scale, collaborative effort to estimate the reproducibility of psychological science. Perspectives on Psychological Science, 7, 657–660.
38.
OswaldF. L.MitchellG.BlantonH.JaccardJ.TetlockP. E. (2013). Predicting ethnic and racial discrimination: A meta-analysis of IAT criterion studies. Journal of Personality and Social Psychology, 105, 171–192.
39.
RosenthalR. (1966). Experimenter effects in behavioral research. East Norwalk, CT: Appleton-Century-Crofts.
40.
SatopääV. A.BaronJ.FosterD. P.MellersB. A.TetlockP. E.UngarL. H. (2014). Combining multiple probability predictions using a simple logit model. International Journal of Forecasting, 30, 344–356.
41.
SatopääV. A.JensenS. T.MellersB. A.TetlockP. E.UngarL. H. (in press). Probability aggregation in time-series: Dynamic hierarchical modeling of sparse expert beliefs. Annals of Applied Statistics.
42.
SimmonsJ. P.NelsonL. D.SimonsohnU. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22, 1359–1366.
43.
SteyversM.WallstenT. S.MerkleE. C.TurnerB. M. (2014). Evaluating probabilistic forecasts with Bayesian signal detection models. Risk Analysis, 34, 435–452.
44.
StoneE.LuuJ. (2012, November). Improving forecaster accuracy using automatized calibration training experts. Poster session presented at the 33rd Annual Conference of the Society for Judgment and Decision Making, Minneapolis, MN.
45.
TetlockP. E. (2005). Expert political judgment: How good is it? How can we know?Princeton, NJ: Princeton University Press.
46.
TetlockP. E.MansteadA. S. (1985). Impression management versus intrapsychic explanations in social psychology: A useful dichotomy?Psychological Review, 92, 59–77.
47.
TetlockP. E.MellersB. A. (2011). Intelligent management of intelligence agencies: Beyond accountability ping-pong. American Psychologist, 66, 542–554.
48.
TetlockP. E.MitchellG. (2009). Implicit bias and accountability systems: What must organizations do to prevent discrimination?Research in Organizational Behavior, 29, 3–38.
49.
TurnerB. M.SteyversM.MerkleE. C.BudescuD. V.WallstenT. S. (2013). Forecast aggregation via recalibration. Machine Learning. Advance online publication. doi:10.1007/s10994-013-5401-4
50.
WallstenT. S.BudescuD. V.ZwickR. (1993). Comparing the calibration and coherence of numerical and verbal probability judgments. Management Science, 39, 176–190
