The wisdom of the crowd is the finding that aggregating the judgments of many people can lead to surprisingly accurate group judgments. Usually statistical methods are used to aggregate people’s judgments, but there are advantages to using cognitive models instead. Crowd judgments based on cognitive modeling can (a) identify experts and amplify their judgments, (b) provide a representational structure for aggregating complicated multidimensional judgments, (c) debias judgments that are affected by heuristic cognitive processes or competitive social situations, and (d) diversify the crowd by incorporating predictions about judgments that have not been observed. Demonstrations of these advantages are provided in case studies involving ranking, probability estimation, and categorization problems.
AndersR.BatchelderW. H. (2012). Cultural consensus theory for multiple consensus truths. Journal of Mathematical Psychology, 56(6), 452–469. https://doi.org/10.1016/j.jmp.2013.01.004
BennettS. T.BenjaminA. S.MistryP. K.SteyversM. (2018). Making a wiser crowd: Benefits of individual metacognitive control on crowd performance. Computational Brain & Behavior, 1, 90–99. https://doi.org/10.1007/s42113-018-0006-4
4.
CavagnaroD. R.PittM. A.GonzalezR.MyungI. J. (2013). Discriminating among probability weighting functions with adaptive design optimization. Journal of Risk and Uncertainty, 47, 255–289. https://doi.org/10.1007/s11166-013-9179-3
5.
ChristianB.GriffithsT. (2016). Algorithms to live by: The computer science of human decisions. Henry Holt & Co.
6.
DanileikoI.LeeM. D. (2017). A model-based approach to the wisdom of the crowd in category learning. Cognitive Science, 42, 861–883. https://doi.org/10.1111/cogs.12561
GordonA. D. (1999). Classification (2nd ed.). Chapman & Hall/CRC Press.
9.
HasanE.EichbaumQ.SeegmillerA. C.StrattonC.TruebloodJ. S. (2022). Improving medical image decision-making by leveraging metacognitive processes and representational similarity. Topics in Cognitive Science, 14(2), 400–413. https://doi.org/10.1111/tops.12588
10.
HasanE.EichbaumQ.SeegmillerA. C.StrattonC.TruebloodJ. S. (2024). Harnessing the wisdom of the confident crowd in medical image decision-making. Decision, 1(11), 127–149. https://doi.org/10.1037/dec0000210
HuttenlocherJ.HedgesL. V.DuncanS. (1991). Categories and particulars: Prototype effects in estimating spatial location. Psychological Review, 98(3), 352–376. https://doi.org/10.1037//0033-295x.98.3.352
13.
LeeM. D.DanileikoI. (2014). Using cognitive models to combine probability estimates. Judgment and Decision Making, 9, 259–273. https://doi.org/10.1017/s1930297500005799
14.
LeeM. D.DanileikoI.ViJ. (2018). Testing the ability of the surprisingly popular method to predict NFL games. Judgment and Decision Making, 13, 322–333. https://doi.org/10.31234/osf.io/4a5bw
15.
LeeM. D.KeM. Y. (2022). Modeling individual differences in beliefs and opinions using Thurstonian models. In MusolinoJ.HemmerP.SommerJ. (Eds.), The science of beliefs (pp. 488–511). Cambridge University Press.
16.
LeeM. D.SteyversM.de YoungM.MillerB. J. (2012). Inferring expertise in knowledge and prediction ranking tasks. Topics in Cognitive Science, 4, 151–163. https://doi.org/10.1111/j.1756-8765.2011.01175.x
LichtensteinS.SlovicP.FischhoffB.LaymanM.CombsB. (1978). Judged frequency of lethal events. Journal of Experimental Psychology: Human Learning and Memory, 4(6), 551–578. https://doi.org/10.1037/0278-7393.4.6.551
21.
MayerM.HeckD. W. (2023). Cultural consensus theory for two-dimensional location judgments. Journal of Mathematical Psychology, 113, Article 102742. https://doi.org/10.1016/j.jmp.2022.102742
22.
McCoyJ.PrelecD. (2023). A Bayesian hierarchical model of crowd wisdom based on predicting opinions of others. Management Science. Advance online publication. https://doi.org/10.1287/mnsc.2023.4955
23.
MontgomeryL. E.BaldiniC. M.VanderkerckhoveJ.LeeM. D. (2024). Where’s Waldo, Ohio? Using cognitive models to improve the aggregation of spatial knowledge. Computational Brain & Behavior, 7, 242–254. https://doi.org/10.1007/s42113-024-00200-0
24.
MontgomeryL. E.BradfordN.LeeM. D. (2024). The wisdom of the crowd with partial rankings: A Bayesian approach implementing the Thurstone model in JAGS. Behavior Research Methods. Advance online publication. https://doi.org/10.3758/s13428-024-02479-0
25.
PrelecD.SeungH. S.McCoyJ. (2017). A solution to the single-question crowd wisdom problem. Nature, 541, 532–535. https://doi.org/10.1038/nature21054
26.
RobbinsH. (1952). Some aspects of the sequential design of experiments. Bulletin of the American Mathematical Society, 55, 527–535. https://doi.org/10.1007/978-1-4612-5110-1_13
SteyversM.LeeM. D.MillerB.HemmerP. (2009). The wisdom of crowds in the recollection of order information. In BengioY.SchuurmansD.LaffertyJ.WilliamsC. K. I.CulottaA. (Eds.), Advances in Neural Information Processing Systems 22 (pp. 1785–1793). Neural Information Processing Systems Foundation, Inc.
29.
SteyversM.MillerB. (2015). Cognition and collective intelligence. In MaloneT. W.BernsteinM. S. (Eds.), Handbook of collective intelligence (pp. 119–138). MIT Press.
30.
SurowieckiJ. (2004). The wisdom of crowds. Random House.
31.
ThomasB.CoonJ.WestfallH. A.LeeM. D. (2021). Model-based wisdom of the crowd for sequential decision-making tasks. Cognitive Science, 45(7), Article e13011. https://doi.org/10.1111/cogs.13011
32.
ThurstoneL. L. (1927). The method of paired comparisons for social values. The Journal of Abnormal and Social Psychology, 21, 384–400. https://doi.org/10.1037/h0065439
van DoornJ.WestfallH. A.LeeM. D. (2021). Using the weighted Kendall’s distance to analyze rank data in psychology. The Quantitative Methods for Psychology, 17, 154–165. https://doi.org/10.20982/tqmp.17.2.p154
