This study manipulates the presence and reliability of AI recommendations for risky decisions to measure the effect on task performance, behavioral consequences of trust, and deviation from a probability matching collaborative decision-making model.
Background
Although AI decision support improves performance, people tend to underutilize AI recommendations, particularly when outcomes are uncertain. As AI reliability increases, task performance improves, largely due to higher rates of compliance (following action recommendations) and reliance (following no-action recommendations).
Methods
In a between-subject design, participants were assigned to a high reliability AI, low reliability AI, or a control condition. Participants decided whether to bet that their team would win in a series of basketball games tying compensation to performance. We evaluated task performance (in accuracy and signal detection terms) and the behavioral consequences of trust (via compliance and reliance).
Results
AI recommendations improved task performance, had limited impact on risk-taking behavior, and were under-valued by participants. Accuracy, sensitivity (d’), and reliance increased in the high reliability AI condition, but there was no effect on response bias (c) or compliance. Participant behavior was only consistent with a probability matching model for compliance in the low reliability condition.
Conclusion
In a pay-off structure that incentivized risk-taking, the primary value of the AI recommendations was in determining when to perform no action (i.e., pass on bets).
Application
In risky contexts, designers need to consider whether action or no-action recommendations will be more influential to design appropriate interventions.
ArnoldV.CollierP. A.LeechS. A.SuttonS. G. (2004). Impact of intelligent decision aids on expert and novice decision-makers’ judgments. Accounting and Finance, 44(1), 1–26. https://doi.org/10.1111/j.1467-629x.2004.00099.x
2.
BartlettM. L.McCarleyJ. S. (2017). Benchmarking aided decision making in a signal detection task. Human Factors: The Journal of the Human Factors and Ergonomics Society, 59(6), 881–900. https://doi.org/10.1177/0018720817700258
3.
BerettaA.ZancanaroM.LepriB. (2019, September) Following wrong suggestions: self-blame in human and computer scenarios. In IFIP Conference on Human-Computer Interaction (pp. 542–550). Springer, Cham.
4.
BoskemperM. M.BartlettM. L.McCarleyJ. S. (2021). Measuring the Efficiency of Automation-Aided Performance in a Simulated Baggage Screening Task. Human Factors. https://doi.org/10.1177/0018720820983632
5.
CanfieldC. I.FischhoffB.DavisA. (2016). Quantifying phishing susceptibility for detection and behavior decisions. Human Factors: The Journal of the Human Factors and Ergonomics Society, 58(8), 1158–1172. https://doi.org/10.1177/0018720816665025
6.
ChanceyE. T.BlissJ. P.YamaniY.HandleyH. A. H. (2017). Trust and the Compliance–Reliance Paradigm: The Effects of Risk, Error Bias, and Reliability on Trust and Dependence. Human Factors: The Journal of the Human Factors and Ergonomics Society, 59(3), 333–345. https://doi.org/10.1177/0018720816682648
7.
ChavaillazA.SchwaningerA.MichelS.SauerJ. (2019). Expertise, Automation and Trust in X-Ray Screening of Cabin Baggage. Frontiers in Psychology, 10(10), 256. https://doi.org/10.3389/fpsyg.2019.00256
8.
ChowE. K. H.MassieA. B.MuzaaleA. D.SingerA. L.KucirkaL. M.MontgomeryR. A.LehmannH. P.SegevD. L. (2013). Identifying Appropriate Recipients for CDC Infectious Risk Donor Kidneys: Recipients for Infectious Risk Donor Kidneys. American Journal of Transplantation, n/a-n/a.https://doi.org/10.1002/ajt.12206
9.
DietvorstB. J.SimmonsJ. P.MasseyC. (2015). Algorithm aversion: People erroneously avoid algorithms after seeing them err. Journal of Experimental Psychology: General, 144(1), 114–126.
10.
DuxburyD.SummersB. (2004). Financial risk perception: Are individuals variance averse or loss averse?Economics Letters, 84(1), 21–28.
11.
DzindoletM. T.BeckH. P.PierceL. G. (2000). Encouraging human operators to appropriately rely on automated decision aids. In Proceedings of the 2000 Command and Control Research and Technology Symposium.
12.
FanX.OhS.McNeeseM.YenJ.CuevasH.StraterL.EndsleyM. R. (2008). The influence of agent reliability on trust in human-agent collaboration. In Proceedings of the 15th European Conference on Cognitive Ergonomics: The Ergo- nomics of Cool Interaction (Article 7). New York, NY: ACM.
13.
FischerI.BudescuD. V. (2005). When do those who know more also know more about how much they know? The development of confidence and performance in categorical decision tasks. Organizational Behavior and Human Decision Processes, 98(1), 39–53. https://doi.org/10.1016/j.obhdp.2005.04.003
14.
GigerenzerG.GaissmaierW.Kurz-MilckeE.SchwartzL. M.WoloshinS. (2007). Helping Doctors and Patients Make Sense of Health Statistics. Psychological Science in the Public Interest, 8(2), 53–96. https://doi.org/10.1111/j.1539-6053.2008.00033.x
15.
GoddardK.RoudsariA.WyattJ. C. (2014). Automation bias: Empirical results assessing influencing factors. International Journal of Medical Informatics, 83(5), 368–375. https://doi.org/10.1016/j.ijmedinf.2014.01.001
16.
HeitE.PriceP. C.BowerG. H. (1994). A model for predicting the outcomes of basketball games. Applied Cognitive Psychology, 8(7), 621–639. https://doi.org/10.1002/acp.2350080703
17.
HoffK. A.BashirM. (2015). Trust in Automation: Integrating Empirical Evidence on Factors That Influence Trust. Human Factors: The Journal of the Human Factors and Ergonomics Society, 57(3), 407–434. https://doi.org/10.1177/0018720814547570
KaplanA. D.KesslerT. T.BrillJ. C.HancockP. A. (2021). Trust in Artificial Intelligence: Meta-Analytic Findings. Human Factors. https://doi.org/10.1177/00187208211013988
20.
KirlikA. (1993). Modeling strategic behavior in human-automation interaction: Why an" aid" can (and should) go unused. Human Factors, 35(2), 221–242. https://doi.org/10.1177/001872089303500203
21.
LangerM.KönigC. J.PapathanasiouM. (2019). Highly automated job interviews: Acceptance under the influence of stakes. International Journal of Selection and Assessment, 27(3), 217–234. https://doi.org/10.1111/ijsa.12246
22.
LeeJ. D.MorayN. (1994). Trust, self-confidence, and operators’ adaptation to automation. International Journal of Human-Computer Studies, 40(1), 153–184. https://doi.org/10.1006/ijhc.1994.1007
23.
LeeJ. D.SeeK. A. (2004). Trust in Automation: Designing for Appropriate Reliance. Human Factors: The Journal of the Human Factors and Ergonomics Society, 46(1), 50–80. https://doi.org/10.1518/hfes.46.1.50_30392
MandrikC.BaoY. (2005). Exploring the Concept and Measurement of General Risk Aversion. Advances in Consumer Research, 32, 531–539.
29.
ManzeyD.GérardN.WiczorekR. (2014). Decision-making and response strategies in interaction with alarms: The impact of alarm reliability, availability of alarm validity information and workload. Ergonomics, 57(12), 1833–1855. https://doi.org/10.1080/00140139.2014.957732
30.
McKinneyS. M.SieniekM.GodboleV.GodwinJ.AntropovaN.AshrafianH.BackT.ChesusM.CorradoG. S.DarziA.EtemadiM.Garcia-VicenteF.GilbertF. J.Halling-BrownM.HassabisD.JansenS.KarthikesalingamA.KellyC. J.KingD.ShettyS. (2020). International evaluation of an AI system for breast cancer screening. Nature, 577(7788), 89–94. https://doi.org/10.1038/s41586-019-1799-6
31.
McKnightD.ChoudhuryV.KacmarC. (2002). Developing and validating trust measures for e-commerce: An integrative typology. Inf. Syst. Res, 13(1), 334–359.
32.
MeyerJ. (2004). Conceptual Issues in the Study of Dynamic Hazard Warnings. Human Factors: The Journal of the Human Factors and Ergonomics Society, 46(2), 196–204. https://doi.org/10.1518/hfes.46.2.196.37335
33.
MeyerJ.KucharJ. K. (2021). Maximal benefits and possible detrimental effects of binary decision aids. In 2021 IEEE 2nd International Conference on Human-Machine Systems (ICHMS) (pp. 1–6). IEEE.
34.
ParasuramanR.RileyV. (1997). Humans and automation: Use, misuse, disuse, abuse. Human Factors: The Journal of the Human Factors and Ergonomics Society, 39(2), 230–253. https://doi.org/10.1518/001872097778543886
35.
PeerE.BrandimarteL.SamatS.AcquistiA. (2017). Beyond the Turk: Alternative platforms for crowdsourcing behavioral research. Journal of Experimental Social Psychology, 70(1), 153–163. https://doi.org/10.1016/j.jesp.2017.01.006
36.
PerkinsL.MillerJ. E.HashemiA.BurnsG. (2010). Designing for human-centered systems: Situational Risk as a Factor of Trust in Automation. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 54(25), 2130–2134. https://doi.org/10.1177/154193121005402502
37.
QuinnD. B. (2018). Exploring the Efficacy of Social Trust Repair in Human-Automation Interactions. 68.
38.
RaschR.KottA.ForbusK. D. (2003). Incorporating AI into military decision making: An experiment. IEEE Intelligent Systems, 18(4), 18–26. https://doi.org/10.1109/MIS.2003.1217624
39.
RiceS.KellerD. (2009). Automation Reliance Under Time Pressure. Cognitive Technology, 14(1), 36–44.
40.
RiegerT.ManzeyD. (2020). Human Performance Consequences of Automated Decision Aids: The Impact of Time Pressure. Human Factors, 1–18. https://doi.org/10.1177/0018720820965019
41.
RieskampJ.HoffrageU. (2008). Inferences under time pressure: How opportunity costs affect strategy selection. Acta Psychologica, 127(2), 258–276. https://doi.org/10.1016/j.actpsy.2007.05.004
StanislawH.TodorovN. (1999). Calculation of signal detection theory measures. Behavior Research Methods, Instruments, & Computers, 31(1), 137–149. https://doi.org/10.3758/BF03207704
45.
SutherlandS. C.HarteveldC.YoungM. E. (2016). Effects of the advisor and environment on requesting and complying with automated advice. ACM Transactions on Interactive Intelligent Systems (TiiS), 6(4), 1–36.
VenkateshV.DavisF. D. (2000). A Theoretical Extension of the Technology Acceptance Model: Four Longitudinal Field Studies. Management Science, 46(2), 186–204. https://doi.org/10.1287/mnsc.46.2.186.11926
49.
VoelklB. (2019). Multiple testing: Correcting for alpha error inflation with false discovery rate (FDR) or family-wise error rate?Animal Behaviour, 155(1), 173–177. https://doi.org/10.1016/j.anbehav.2019.07.001
50.
WangL.JamiesonG. A.HollandsJ. G. (2009). Trust and reliance on an automated combat identification system. Human Factors, 51(1), 281–291.
51.
WickensC. D.DixonS. R. (2007). The benefits of imperfect diagnostic automation: A synthesis of the literature. Theoretical Issues in Ergonomics Science, 8(3), 201–212. https://doi.org/10.1080/14639220500370105
52.
WickensC.XuX. (2002). How Does Automation Reliability Influence Workload ?/paper/How-Does-Automation-Reliability-Influence-Workload-Wickens-Xu/99e3a8dd2a8bf1b7eccf850e63e014da22b5b6d5.
53.
WiczorekR.MeyerJ. (2019). Effects of Trust, Self-Confidence, and Feedback on the Use of Decision Automation. Frontiers in Psychology, 10(1), 519. https://doi.org/10.3389/fpsyg.2019.00519
54.
WiczorekR.MeyerJ.GuenzlerT. (2012). On the Relation Between Reliance and Compliance in an Aided Visual Scanning Task. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 56(1), 253–257. https://doi.org/10.1177/1071181312561060
55.
ZhangY.LiaoQ. V.BellamyR. K. E. (2020). Effect of confidence and explanation on accuracy and trust calibration in AI-assisted decision making. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency, 295–305. https://doi.org/10.1145/3351095.3372852
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
