Drivers’ Understanding of Artificial Intelligence in Automated Driving Systems: A Study of a Malicious Stop Sign

Abstract

Automated Driving Systems (ADS), like many other systems people use today, depend on successful Artificial Intelligence (AI) for safe roadway operations. In ADS, an essential function completed by AI is the computer vision techniques for detecting roadway signs by vehicles. The AI, though, is not always reliable and sometimes requires the human’s intelligence to complete a task. For the human to collaborate with the AI, it is critical to understand the human’s perception of AI. In the present study, we investigated how human drivers perceive the AI’s capabilities in a driving context where a stop sign is compromised and how knowledge, experience, and trust related to AI play a role. We found that participants with more knowledge of AI tended to trust AI more, and those who reported more experience with AI had a greater understanding of AI. Participants correctly deduced that a maliciously manipulated stop sign would be more difficult for AI to identify. Nevertheless, participants still overestimated the AI’s ability to recognize the malicious stop sign. Our findings suggest that the public do not yet have a sufficiently accurate understanding of specific AI systems, which leads them to over-trust the AI in certain conditions.

Keywords

Malicious attack artificial intelligence computer vision artificial intelligence in automated driving systems understanding of artificial intelligence trust in artificial intelligence

Get full access to this article

View all access options for this article.

References

Asan

Bayrak

A. E.

Choudhury

(2020). Artificial intelligence and human trust in healthcare: Focus on clinicians. Journal of Medical Internet Research, 22(6), Article e15154. https://doi.org/10.2196/15154

Bansal

Nushi

Kamar

Lasecki

W. S.

Weld

D. S.

Horvitz

(2019). Beyond accuracy: The role of mental models in human-AI team performance. Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, 7(1), 2–11. https://ojs.aaai.org/index.php/HCOMP/article/view/5285

Cunneen

Mullins

Murphy

(2019). Autonomous vehicles and embedded artificial intelligence: The challenges of framing machine driving decisions. Applied Artificial Intelligence, 33(8), 706–731. https://doi.org/10.1080/08839514.2019.1600301

Eykholt

Evtimov

Fernandes

Rahmati

Xiao

Prakash

Kohno

Song

(2017). Robust physical-world attacks on deep learning models. https://doi.org/10.48550/arXiv.1707.08945

Ferrario

Loi

Viganò

(2021). Trust does not need to be human: It is possible to trust medical AI. Journal of Medical Ethics, 47(6), 437–438. https://doi.org/10.1136/medethics-2020-106922

Goodfellow

I. J.

Shlens

Szegedy

(2014). Explaining and harnessing adversarial examples. https://doi.org/10.48550/arXiv.1412.6572

Grace

Salvatier

Dafoe

Zhang

Evans

(2018). When will AI exceed human performance? Evidence from AI experts. Journal of Artificial Intelligence Research, 62, 729–754. https://doi.org/10.1613/jair.1.11222

Halim

Kalsoom

Bashir

Abbas

(2016). Artificial intelligence techniques for driving safety and vehicle crash prediction. Artificial Intelligence Review, 46(3), 351–387. https://doi.org/10.1007/s10462-016-9467-9

Hengstler

Enkel

Duelli

(2016). Applied artificial intelligence and trust-The case of autonomous vehicles and medical assistance devices. Technological Forecasting and Social Change, 105, 105–120. https://doi.org/10.1016/j.techfore.2015.12.014

10.

Hoff

K. A.

Bashir

(2015). Trust in automation: Integrating empirical evidence on factors that influence trust. Human Factors, 57(3), 407–434. https://doi.org/10.1177/0018720814547570

11.

Holzinger

Searle

Wernbacher

(2011). The effect of previous exposure to technology on acceptance and its importance in usability and accessibility engineering. Universal Access in the Information Society, 10(3), 245–260. https://doi.org/10.1007/s10209-010-0212-x

12.

Jacovi

Marasović

Miller

Goldberg

(2021). Formalizing trust in artificial intelligence: Prerequisites, causes and goals of human trust in AI. Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, 624–635. https://doi.org/10.1145/3442188.3445923

13.

Jarrahi

M. H.

(2018). Artificial intelligence and the future of work: Human-AI symbiosis in organizational decision making. Business Horizons, 61(4), 577–586. https://doi.org/10.1016/j.bushor.2018.03.007

14.

Jian

J.-Y.

Bisantz

A. M.

Drury

C. G.

(2000). Foundations for an empirically determined scale of trust in automated systems. International Journal of Cognitive Ergonomics, 4(1), 53–71. https://doi.org/10.1207/S15327566IJCE0401_04

15.

Kaplan

Haenlein

(2019). Siri, Siri, in my hand: Who’s the fairest in the land? On the interpretations, illustrations, and implications of artificial intelligence. Business Horizons, 62(1), 15–25. https://doi.org/10.1016/j.bushor.2018.08.004

16.

Kendall

Gal

(2017). What uncertainties do we need in bayesian deep learning for computer vision? Proceedings of Advances in Neural Information Processing Systems. arXiv preprint arXiv1703.04977.

17.

Kyriakidis

Happee

De Winter

J. C. F.

(2015). Public opinion on automated driving: Results of an international questionnaire among 5000 respondents. Transportation Research Part F: Traffic Psychology and Behaviour, 32, 127–140. https://doi.org/10.1016/j.trf.2015.04.014

18.

Lee

J. D.

See

K. A.

(2004). Trust in automation: Designing for appropriate reliance. Human Factors, 46(1), 50–80. https://doi.org/10.1518/hfes.46.1.50_30392

19.

Liu

Wang

Fan

Liu

Jain

Liu

Jain

A. K.

Tang

(2021). Trustworthy AI: A computational perspective. https://doi.org/10.48550/arXiv.2107.06641

20.

Lyons

J. B.

Guznov

S. Y.

(2019). Individual differences in human-machine trust: A multi-study look at the perfect automation schema. Theoretical Issues in Ergonomics Science, 20(4), 440–458. https://doi.org/10.1080/1463922X.2018.1491071

21.

Wang

Yang

(2020). Artificial intelligence applications in the development of autonomous vehicles: A survey. IEEE/CAA Journal of Automatica Sinica, 7(2), 315–329. https://doi.org/10.1109/JAS.2020.1003021

22.

Moore

D. A.

Healy

P. J.

(2008). The trouble with overconfidence. Psychological Review, 115(2), 502–517. https://doi.org/10.1037/0033-295X.115.2.502

23.

Lee

Kim

Park

S. H.

Kwon

Suh

(2017). Us vs. them: Understanding artificial intelligence technophobia over the Google DeepMind Challenge match. Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems, 2523–2534. https://doi.org/10.1145/3025453.3025539

24.

Song

Choi

Kim

Lee

Suh

(2018). I lead, you help but only with enough details: Understanding the user experience of co-creation with artificial intelligence. Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, 1–13. https://doi.org/10.1145/3173574.3174223

25.

Parasuraman

Riley

(1997). Humans and automation: Use, misuse, disuse, abuse. Human Factors, 39(2), 230–253. https://doi.org/10.1518/001872097778543886

26.

Prahl

Goh

W. W. P.

(2021). “Rogue machines” and crisis communication: When AI fails, how do companies publicly respond? Public Relations Review, 47(4), 102077. https://doi.org/10.1016/j.pubrev.2021.102077

27.

Qiu

Liu

Zhou

(2019). Review of artificial intelligence adversarial attack and defense technologices. Applied Sciences, 9(5), 909. https://doi.org/10.3390/app9050909

28.

Reig

Norman

Morales

C. G.

Das

Steinfeld

Forlizzi

(2018). A field study of pedestrians and autonomous vehicles. Proceedings of the 10th International Conference on Automotive User Interfaces and Interactive Vehicular Applications, 198–209. https://doi.org/10.1145/3239060.3239064

29.

Rossi

(2018). Building trust in artificial intelligence. Journal of International Affairs, 72(1), 127–134. https://www.jstor.org/stable/26588348

30.

Röttger

Bali

Manzey

(2009). Impact of automated decision aids on performance, operator behaviour and workload in a simulated supervisory control task. Ergonomics, 52(5), 512–523. https://doi.org/10.1080/00140130802379129

31.

Russell

Moskowitz

I. S.

Raglin

(2017). Human information interaction, artificial intelligence, and errors. In Lawless

W. F.

Mittu

Sofge

, et al. (Eds.), Autonomy and artificial intelligence: A threat or savior? (pp. 71–101). Springer. https://doi.org/10.1007/978-3-319-59719-5_4

32.

Sabottke

C. F.

Spieler

B. M.

(2020). The effect of image resolution on deep learning in radiography. Radiology: Artificial Intelligence, 2(1), Article e190015. https://doi.org/10.1148/ryai.2019190015

33.

SAE International . (2021). Taxonomy and definitions for terms related to driving automation systems for on-road motor vehicles (Publication No. J3016_202104). https://doi.org/10.4271/J3016_202104

34.

Sanchez

Fisk

A. D.

Rogers

W. A.

(2004). Reliability and age-related effects on trust and reliance of a decision support aid. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 48(3), 586–589. https://doi.org/10.1177/154193120404800366

35.

Shin

(2021). The effects of explainability and causability on perception, trust, and acceptance: Implications for explainable AI. International Journal of Human-Computer Studies, 146, 102551. https://doi.org/10.1016/j.ijhcs.2020.102551

36.

Siau

Wang

(2018). Building trust in artificial intelligence, machine learning, and robotics. Cutter Business Technology Journal, 31(2), 47–53. https://www.cutter.com/article/building-trust-artificial-intelligence-machine-learning-and-robotics-498981

37.

Tenhundfeld

N. L.

de Visser

E. J.

Ries

A. J.

Finomore

V. S.

Tossell

C. C.

(2020). Trust and distrust of automated parking in a Tesla Model X. Human Factors, 62(2), 194–210. https://doi.org/10.1177/0018720819865412

38.

Tschopp

(2020, May 7). PAS - The perfect automation schema. Influencing Trust. https://www.scip.ch/en/?labs.20200507

39.

Wiegmann

D. A.

Rich

Zhang

(2001). Automated diagnostic aids: The effects of aid reliability on users’ trust and reliance. Theoretical Issues in Ergonomics Science, 2(4), 352–367. https://doi.org/10.1080/14639220110110306

40.

Guo

Chao

W.-L.

Weinberger

K. Q.

(2019). A new defense against adversarial images: Turning a weakness into a strength. Proceedings of the 33^rd Conference on Neural Information Processing Systems, 4(5), 1635–1646. https://doi.org/10.48550/arXiv.1910.07629

41.

Zhang

Gao

Guo

Liu

(2016). A study on key technologies of unmanned driving. CAAI Transactions on Intelligence Technology, 1(1), 4–13. https://doi.org/10.1016/j.trit.2016.03.003