Systemic Trust in Artificial Intelligence

Abstract

Engineering more trustworthy AI is increasingly important as AI is being applied to more high-consequence work. We argue that trust in AI is often examined too narrowly, and that there is value in applying more naturalistic methods to examine trust in the context of the entire sociotechnical system (i.e., people, technology, and work). We present a case study where we interviewed submariners, and describe how interactions among the people, technology, and work in submarine operations can affect trust in AI. We also discuss the practical benefits of trust as a social contract between end users and developers of AI.

Keywords

Artificial Intelligence (AI)trust naturalistic decision making systems engineering

Get full access to this article

View all access options for this article.

References

Ala-Luopa

Olsson

Väänänen

Hartikainen

Makkonen

(2024). Trusting intelligent automation in expert work: Accounting practitioners’ experiences and perceptions. Computer Supported Cooperative Work, 33, 1343–1371. https://doi.org/10.1007/s10606-024-09499-6

Andras

Esterle

Guckert

Han

T. H.

Lewis

P. R.

Milanovic

Payne

Perret

Pitt

Powers

S. T.

Urquhart

Wells

(2018). Trusting intelligent machines: Deepening trust within socio-technical systems. IEEE Technology and Society Magazine, 37(4), 76–83. https://doi.org/10.1109/MTS.2018.2876107

Atchley

O’Hear

Barr

H. M.

Cotter

J. E.

Hamblin

Oswald

Mesmer

Weger

Gholston

Menon

Demir

Tenhundfeld

N. L.

(2023). How do blame attributions impact trust in complex task environments. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 67(1), 495–501. https://doi.org/10.1177/21695067231192618

Bettin

Steelman

Wallace

Pontious

Veinott

(2022). Identifying and addressing risks in the early design of a sociotechnical system through premortem. Proceedings of the 66th Annual Meeting of the Human Factors and Ergonomics Society, 66(1), 1514–1518. https://doi.org/10.1177/107118132266130

Bierly

P. E.

Spender

J. C.

(1995). Culture and high reliability organizations: The case of the nuclear submarine. Journal of Management, 21(4), 639–656. https://doi.org/10.1016/0149-2063(95)90003-9

Bolton

M. L.

(2022). Trust is not a virtue: Why we should not trust trust. Ergonomics in Design: The Quarterly of Human Factors Applications, 32(4), 4–11. https://doi.org/10.1177/10648046221130171

Bolton

M. L.

Hancock

P. A.

Lee

J. D.

Montague

Yang

X. J.

(2023). Does trust have value? A discussion about the importance of trust to human factors and engineering. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 67(1), 137–138. https://doi.org/10.1177/21695067231192235

Browne

J. T.

Bakker

Lloyd

Ben Allouch

(2022). Trust in clinical AI: Expanding the unit of analysis. Frontiers in Artificial Intelligence and Applications, 354, 96–113. https://doi.org/10.3233/FAIA220192

Bruni

Freiman

Riddle

(2023). Beyond the tool vs. teammate debate: Exploring the sidekick metaphor in designing human-AI dyads. Human Factors and Simulation, 83, 12–24. https://doi.org/10.54941/ahfe1003558

10.

Cho

J. Y.

Lee

(2014). Reducing confusion about grounded theory and qualitative content analysis: Similarities and differences. Qualitative Report, 19(64), 1–20. https://doi.org/10.46743/2160-3715/2014.1028

11.

Clegg

C. W.

(2000). Sociotechnical principles for system design. Applied Ergonomics, 31(5), 463–477. https://doi.org/10.1016/S0003-6870(00)00009-0

12.

Cordle

(2018). Design systems that work for people. US Naval Institute Proceedings, 144(9). https://usni.org/magazines/proceedings/2018/september/design-systems-work-people

13.

Dorton

S. L.

(2022). Supradyadic trust in artificial intelligence. AHFE International, 28, 92–100. https://doi.org/10.54941/ahfe1001451

14.

Dorton

S. L.

Harper

(2021). Trustable AI: A critical challenge for naval intelligence. Center for international maritime security (CIMSEC). https://cimsec.org/trustable-ai-a-critical-challenge-for-naval-intelligence/

15.

Dorton

S. L.

Harper

S. B.

(2022). A naturalistic investigation of trust, AI, and intelligence work. Journal of Cognitive Engineering and Decision Making, 16(4), 222–236. https://doi.org/10.1177/15553434221103718

16.

Dorton

S. L.

Ministero

L. M.

Alaybek

Bryant

D. J.

(2023). Foresight for ethical AI. Frontiers in artificial intelligence, 6, Article 1143907. https://doi.org/10.3389/frai.2023.1143907

17.

Dorton

S. L.

Stanley

J. C.

(2024). Minding the gap: Tools for trust engineering of artificial intelligence. Ergonomics in Design: The Quarterly of Human Factors Applications, 33(3), 142–147. https://doi.org/10.1177/10648046241249903

18.

Ezer

Bruni

Cai

Hepenstal

S. J.

Miller

C. A.

Schmorrow

D. D.

(2019). Trust engineering for human-AI teams. Proceedings of the Human Factors and Ergonomics Society, 63(1), 322–326. https://doi.org/10.1177.1071181319631264

19.

Henrique

B. M.

Santos

(2024). Trust in artificial intelligence: Literature review and main path analysis. Computers in Human Behavior: Artificial Humans, 2(1), 1–13. https://doi.org/10.1016/j.chbah.2024.100043

20.

Sadler

G. G.

Hoffman

L. C.

Zemlicka

Lyons

Fergueson

Richardson

Cacanindin

Cals

Wilkins

(2017). A longitudinal field study of Auto-GCAS acceptance and trust: First-year results and implications. Journal of Cognitive Engineering and Decision Making, 11(3), 239–251. https://doi.org/10.1177/1555343417701019

21.

Hoff

K. A.

Bashir

(2015). Trust in automation: Integrating empirical evidence on factors that influence trust. Human Factors, 57(3), 407–434. https://doi.org/10.1177/001872081454757

22.

Huang

Cooke

N. J.

Gutzwiller

R. S.

Berman

Chiou

E. K.

Demir

Zhang

(2021). Distributed dynamic team trust in human, artificial intelligence, and robot teaming. Trust in Human-Robot Interaction, 2017, 301–319. https://doi.org/10.1016/B978-0-12-819472-0.00013-7

23.

Johnson

(2019). Artificial intelligence – An enabler of naval tactical decision superiority. AI Magazine, 40(1), 63–78. https://doi.org/10.1609/aimag.v40i1.2852

24.

Johnson

R. B.

(1997). Examining the validity structure of qualitative research. Education, 118(2), 282–292.

25.

Keller

Rice

(2009). System-wide versus component-specific trust using multiple aids. The Journal of General Psychology, 137(1), 114–128. https://doi.org/10.1080/00221300903266713

26.

Kim

S. S. Y.

Watkins

E. A.

Russakovsky

Fong

Monroy Hernandex

(2023). Humans, AI, and context: Understanding end-users’ trust in a real-world computer vision application. 2023 ACM conference on fairness, accountability, and transparency (pp. 77–88). https://doi.org/10.1145/3593013.3593978

27.

Klein

(2007). Performing a project premortem. Harvard Business Review, 85, 18–19.

28.

Landreth

Pfau

(2021). Forging the apex predator: Unmanned systems and SSN(X). Center for International Maritime Security. https://cimsec.org/forging-the-apex-predator-unmanned-systems-and-ssnx/

29.

Landreth

Pfau

(2022). Designing SSN(X) will require modern digital modeling. US Naval Institute Proceedings, 148(10). https://usni.org/magazines/proceedings/2022/october/designing-ssnx-will-require-modern-digital-modeling

30.

Lee

J. D.

See

K. A.

(2004). Trust in automation: Designing for appropriate reliance. Human Factors, 46(1), 50–80. https://doi.org/10.1518/hfes.46.1.50_30392

31.

Lyons

J. B.

Mator

J. D.

Orr

Alarcon

G. M.

Barrera

(2024). Is the pull-down effect overstated? An examination of trust propagation among fighter plots in a high-fidelity simulation. Journal of Cognitive Engineering and Decision Making, 18(2), 99–113. https://doi.org/10.1177/15553434231225909

32.

Montague

Asan

Chiou

(2013). Organizational and technological correlates of nurses’ trust in a smart intravenous pump. Computers, informatics, nursing: CIN, 31(3), 142–149. https://doi.org/10.1097/NXN.0b013e3182812d95

33.

Montague

E. N. H.

Winchester

W. W.

Kleiner

B. M.

(2010). Trust in medical technology by patients and health care providers in obstetric work systems. Behaviour & Information Technology, 29(5), 541–554. https://doi.org/10.1080/01449291003752914

34.

Morrison

B. W.

Bergin

Kelson

Morrison

N. M. V.

Innes

J. M.

Zelic

Al-Saggaf

Paul

(2023). Decision support systems (DSSs) ‘in the wild’: The factors that influence user’ acceptance of DSSs in naturalistic settings. Journal of Cognitive Engineering and Decision Making, 17(4), 332–350. https://doi.org/10.1177/15553434231191385

35.

Murray

A. J.

(1985). The application of artificial intelligence to future tactical C2 design. Naval Engineers Journal, 97(4), 287–295. https://doi.org/10.1111/j.1559-3584.1985.tb01363.x

36.

O’Hear

E. H.

Atchley

Gholston

Weger

Mesmer

Tenhundfeld

N. L.

(2022). System-wide trust: The impact of an error in a multi-component system. Proceedings of the 2022 HFES 66^th International Annual Meeting, 66(1), 1777–1781. https://doi.org/10.1177/107118132266119

37.

Polson

P. G.

Lewis

Wharton

(1992). Cognitive walkthroughs: A method for theory-based evaluation of user interfaces. International Journal of Man-Machine Studies, 36(5), 741–773. https://doi.org/10.1016/0020-7373(92)90039-n

38.

Salwei

M. E.

Carayon

(2022). A sociotechnical systems framework for the application of artificial intelligence in health care delivery. Journal of Cognitive Engineering and Decision Making, 16(4), 194–206. https://doi.org/10.1177/15553434221097357

39.

Thornton

Knowles

Blair

(2022). The alchemy of trust: The creative act of designing trustworthy socio-technical systems. 2022 ACM conference on fairness, accountability, and transparency (pp. 1387–1398). https://doi.org/10.1145/3531146.3533196

40.

Walliser

J. C.

de Visser

E. J.

Shaw

T. H.

(2023). Exploring system wide trust prevalence and mitigation strategies with multiple autonomous agents. Computers in Human Behavior, 143, 1–13, 107671. https://doi.org/10.1016/j.chb.2023.107671

41.

Wong

J. H.

Chiou

E. K.

Gutzwiller

R. S.

Cook

M. B.

Fallon

C. K.

(2024). Human-artificial intelligence teaming for the U.S. Navy: Developing a holistic research roadmap. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 68(1), 380–385. https://doi.org/10.1177/10711813241260352