Sage Journals: Discover world-class research

Abstract

Artificial Intelligence (AI) is becoming ubiquitous in national security work (intelligence, defense, etc.); however, introducing AI into work systems is fraught with challenges. Trust is gained and lost through experiences, and there are many factors that affect trust in AI. Similarly, users adapt their workflows based on trust in these systems. We used a naturalistic approach to understand how intelligence professionals adapted their work practices after gaining or losing trust in AI. We found a variety of adaptations, which were characterized as either being task-based or frequency-based; where users added or removed tasks from their workflow or where they changed the frequency in which they used the AI in their workflow, respectively. We provide specific examples and quotes from participants along with findings, and discuss potential methodological implications for studying and designing AI-driven work systems.

Get full access to this article

View all access options for this article.

References

Diakopoulos

(2016). Accountability in algorithmic decision making. Communications of the ACM, 59(2), 56-62. https://doi.org/10.1145/2844110

Dominguez

Rayo

Dorton

Morey

Naikar

Roth

(2021). Cognitive engineering: Will they know our name when we are 40? Proceedings of the 2021 International Annual Meeting of the Human Factors and Ergonomics Society, 65(1), 257-261. https://doi.org/10.1177/1071181321651042

Dorton

S. L.

Harper

(2021). Trustable AI: A critical challenge for naval intelligence. Center for International and Maritime Security (CIMSEC).

Dorton

S.L.

Harper

S.H.

(2022). A naturalistic investigation of trust, AI, and intelligence work. Journal of Cognitive Engineering and Decision Making. https://doi.org/10.1177/15553434221103718

Flanagan

J. C.

(1954). The critical incident technique. Psychological Bulletin, 5, 327-358. http://dx.doi.org/10.1037/h0061470

Hoff

K. A.

Bashir

(2015). Trust in automation: Integrating empirical evidence on factors that influence trust. Human Factors, 57(3), 407-434. https://doi.org/10.1177/0018720814547570

Hoffman

R.R.

Deal

S.V.

Potter

Roth

E.M.

(2010). The practitioner’s cycles, part 2: Solving envisioned world problems. IEEE Intelligent Systems, 25(3), 6-11. https://doi.org/10.1109/MIS.2010.89

Hoffman

Klein

Muller

S.T.

(2018) Explaining Explanation for “explainable AI”. Proceedings of the Human Factors and Ergonomics Society 2018 Annual Meeting, 62(1), 197-201. https://doi.org/10.1177/1541931218621047

Klein

Jarosz

(2011). A naturalistic study of insight. Journal of Cognitive Engineering and Decision Making, 5(4), 335-351. https://doi.org/10.1177/1555343411427013

10.

Lee

See

K.A.

(2004). Trust in automation: Designing for appropriate reliance. Human Factors, 46(1), 50-80. https://doi.org/10.1518/hfes.46.1.50_30392

11.

Lewicki

R.J.

Wiethoff

(2000). Trust, trust development, and trust repair. In Deutsch

Coleman

P.T.

(Eds.), The Handbook of Conflict Resolution: Theory and Practice, pp. 86-107. Jossey-Bass.

12.

Lyons

J.B.

Ho.

N.T.

Koltai

K.S.

Masequesmay

Skoog

Cacanindin

Johnson

W.J.

(2016). Trust-based analysis of an Air Force collision avoidance system. Ergonomics in Design, 24(1), 9-12. https://doi.org/10.1177/1064804615611274

13.

Papautsky

E.L.

Strouse

Dominguez

(2020). Combining cognitive task analysis and participatory design methods to elicit and represent task flows. Journal of Cognitive Engineering and Decision Making, 14(4), 288-301. https://doi.org/10.1177/1555343420976014

14.

Patterson

E. S.

Rogers

M. L.

Render

M. L.

(2004). A simulation-based embedded probe technique for human-computer interaction evaluation. Cognition, Technology & Work, 6(3), 197-205. https://doi.org/10.1007/s10111004-0161-3

15.

Schaefer

K. E.

Chen

J. Y. C.

Szalma

J. L.

Hancock

P. A.

(2016). A meta-analysis of factors influencing the development of trust in automation: Implications for understanding autonomy in future systems. Human Factors, 58(3), 377-400. https://doi.org/10.1177/0018720816634228

16.

Sherwood

S. M.

Neville

K. J.

McLean

A. L. M. T.

Walwanis

M. M.

Bolton

A. E.

(2020). Integrating new technology into the complex system of air combat training. In Handley

H. A. H

Tolk

(Eds). A framework of human systems engineering: Applications and case studies, 185-204. Wiley. https://doi.org/10.1002/9781119698821.ch10

17.

Symon

P. B.

Tarapore

(2015). Defense intelligence analysis in the age of big data. Joint Forces Quarterly, 79(4), 4-11.

18.

Trent

Hofmann

R.R.

Merritt

Smith

(2019). Modeling the cognitive work of cyber protection teams. The Cyber Defense Review, 4(1), 125-135.

19.

Woods

D.D.

(1998). Designs are hypotheses about how artifacts shape cognitions. Ergonomics, 41, 168-173. https://doi.org/10.1080/001401398187215

Adaptations to Trust Incidents with Artificial Intelligence

Abstract

Get full access to this article

References