Users form snap judgments about decision‑support systems (DSS) that guide later reliance. We examined how those judgments depend on DSS source—machine‑learning algorithm (ML) versus wisdom‑of‑the‑crowd (WoC)—and data scale (small, medium, large). Seventy‑eight participants repeatedly chose between paired advisers (e.g., medium‑ML vs. large‑WoC) and then performed a six‑trial visual‑search task with help from their selected advisor. Participants showed no inherent bias for ML or WoC but consistently preferred advisers built on larger datasets. Within six trials, reliability estimates were calibrated to observed accuracy, indicating rapid trust adjustment. Findings suggest users intuitively weighed sample size over provenance when forecasting DSS performance and that well‑sized crowd systems can compete with contemporary ML advisers.
ChungH.YangX. J. (2024). How do humans learn about the reliability of automation?Cognitive Research: Principles and Implications, 9, 48.
2.
DietvorstB. J.SimmonsJ. P.MasseyC. (2015). Algorithm aversion: People erroneously avoid algorithms after seeing them err. Journal of Experimental Psychology: General, 144(1), 114–126.
3.
GoddardK.RoudsariA.WyattJ. C. (2012). Automation bias: A systematic review of frequency, effect mediators, and mitigators. Journal of the American Medical Informatics Association, 19(1), 121–127.
4.
KaplanJ.McCandlishS.HenighanT.BrownT.ChessB.ChildR.GrayS.RadfordA.WuJ.AmodeiD. (2020). Scaling laws for neural language models. arXiv Preprint arXiv:2001.08361.
5.
LarrickR. P.MannesA. E.SollJ. B. (2012). The social psychology of the wisdom of crowds. In ShermanJ. W. (Ed.), Social judgment and decision making (pp. 227–242). Psychology Press.
6.
LoggJ. M.MinsonJ. A.MooreD. A. (2019). Algorithm appreciation: People prefer algorithmic to human judgment. Organizational Behavior and Human Decision Processes, 151, 90–103.
7.
MadhavanP.WiegmannD. A. (2007). Effects of information source, pedigree, and reliability on operator interaction with decision support systems. Human Factors, 49(5), 773–785.
8.
MedarR.RajpurohitV. S.RashmiB. (2017, August). Impact of training and testing data splits on accuracy of time‑series forecasting in machine learning. In 2017 International conference on computing, communication, control and automation (ICCUBEA) (pp. 1–6). IEEE.
9.
MorewedgeC. K. (2022). Preference for human, not algorithm aversion. Trends in Cognitive Sciences, 26(10), 849–859.
10.
PattonC.WardenA. C.PharmerR. (2023). I Spy differences in system reliability. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 67(1), 940–946.
11.
PearsonC. J.GedenM.MayhornC. B. (2019). Who’s the real expert here? Pedigree’s unique bias on trust between human and automated advisers. Applied Ergonomics, 81, Article 102907.
12.
PopV. L.ShrewsburyA.DursoF. T. (2015). Individual differences in the calibration of trust in automation. Human Factors, 57(4), 545–556.
13.
RiegerT.KuglerL.ManzeyD.RoeslerE. (2024). The (im)perfect automation schema: Who is trusted more, automated or human decision support?Human Factors, 66(8), 1995–2007.
14.
SimoiuC.SumanthC.MysoreA.GoelS. (2019). Studying the “wisdom of crowds” at scale [Conference session]. Proceedings of the AAAI conference on human computation and crowdsourcing, Pittsburgh, PI, USA, Vol. 7, pp. 171–179.
15.
SunC.ShrivastavaA.SinghS.GuptaA. (2017). Revisiting unreasonable effectiveness of data in the deep learning era [Conference session]. Proceedings of the IEEE international conference on computer vision, Cambridge, MA, USA, pp. 843–852.
16.
SurowieckiJ. (2005). The wisdom of crowds. Vintage.
17.
YangX. J.SchemanskeC.GuoY. (2023). From trust to trust dynamics: Combining empirical and computational approaches to model and predict trust in human–autonomy interaction. In Proceedings of the 2023 CHI conference on human factors in computing systems (pp. 1–12). ACM.
18.
YasminR.HassanM. M.GrasselJ. T.BhogarajuH.EscobedoA. R.FuentesO. (2022). Improving crowdsourcing‑based image classification through expanded input elicitation and machine learning. Frontiers in Artificial Intelligence, 5, Article 848056.
19.
YooY.EscobedoA. R.KemmerR.ChiouE. (2024). Elicitation and aggregation of multimodal estimates improve wisdom of crowd effects on ordering tasks. Scientific Reports, 14(1), Article 2640.
