Failing to Grasp our Failure to Grasp Automation Failure

Abstract

This paper discusses three points inspired by Skraaning and Jamieson’s perspective on automation failure: (a) the limitations of the automation failure concept with expanding system boundaries; (b) parallels between the failure to grasp automation failure and the failure to grasp trust in automation; (c) benefits of taking a pluralistic approach to definitions in sociotechnical systems science. While a taxonomy of automation-involved failures may not directly improve our understanding of how to prevent those failures, it could be instrumental for identifying hazards during test and evaluation of operational systems.

Keywords

safety trust first principles definitions models sociotechnical

Get full access to this article

View all access options for this article.

References

Ackoff

R. L.

(1979). The future of operational research is past. Journal of the Operational Research Society, 30(2), 93–104, https://doi.org/10.2307/3009290

Albanowski

Burdick

K. J.

Bonafide

C. P.

Kleinpell

Schlesinger

J. J.

(2023). Ten years later, alarm fatigue is still a safety concern. AACN Advanced Critical Care, 34(3), 189–197. https://doi.org/10.4037/aacnacc2023662

Alsaid

Chiou

E. K.

Lee

J. D.

(2023). Measuring trust: A text analysis approach to compare, contrast, and select trust questionnaires. Frontiers in Psychology, 14(1192020). https://doi.org/10.3389/fpsyg.2023.1192020

Billeaud

Snow

(2023, July 28). The backup driver in the 1st death by a fully autonomous car pleads guilty to endangerment. AP News. https://apnews.com/article/autonomous-vehicle-death-uber-charge-backup-driver-1c711426a9cf020d3662c47c0dd64e35

Blasch

Sung

Nguyen

(2020). Multisource AI scorecard table for system evaluation. In AAAI FSS-20: Artificial Intelligence in Government and Public Sector. ArXiv. https://arxiv.org/abs/2102.03985

Chiou

E. K.

Lee

J. D.

(2021). Trusting automation: Designing for responsivity and resilience. Human Factors, 65(1), 137–165. https://doi.org/10.1177/00187208211009995

Chiou

E. K.

Salehi

Blasch

Sung

Cohen

M. C.

Pan

Mancenido

Mosallanezhad

Bhatti

(2022). Trust in AI-enabled decision support systems: Preliminary validation of MAST criteria. In ICHMS22: 3rd IEEE International Conference on Human-Machine Systems, Orlando, FL, November 17–19, 2022. https://doi.org/10.1109/ICHMS56717.2022.9980623

Cummings

(2018). Informing autonomous system design through the lens of skill-rule-and knowledge-based behaviors. Journal of Cognitive Engineering and Decision Making, 12(1), 58–61. https://doi.org/10.1177/1555343417736461

Cvach

(2012). Monitor alarm fatigue: An integrative review. Biomedical Instrumentation & Technology, 46(4), 268–277. https://doi.org/10.2345/0899-8205-46.4.268

10.

Hancock

P. A.

Billings

D. R.

Schaefer

K. E.

Chen

J. Y. C.

de Visser

E. J.

Parasuraman

(2011). A meta-analysis of factors affecting trust in human-robot interaction. Human Factors, 53(5), 517–527. https://doi.org/10.1177/0018720811417254

11.

Hoff

Bashir

(2013). A theoretical model for trust in automated systems. In CHI ’13 Extended Abstracts on Human Factors in Computing Systems on - CHI EA ’13 (Vol. 115). Association for Computing Machinery.

12.

Huang

Cooke

N. J.

Gutzwiller

R. S.

Berman

Chiou

E. K.

Demir

Zhang

(2021) Distributed dynamic team trust in human, artificial intelligence, and robot teaming. In Nam

C. S.

Lyons

J. B.

(Eds.), Trust in human-robot interaction (pp. 301–319). Academic Press. https://doi.org/10.1016/B978-0-12-819472-0.00013-7

13.

Kohn

S. C.

de Visser

E. J.

Wiese

Lee

Y.-C.

Shaw

T. H.

(2021). Measurement of trust in automation: A narrative review and reference guide. Frontiers in Psychology, 12(604977). https://doi.org/10.3389/fpsyg.2021.604977

14.

Lee

J. D.

See

K. A.

(2004). Trust in automation: Designing for appropriate reliance. Human Factors, 46(1), 50–80. https://doi.org/10.1518/hfes.46.1.50_30392

15.

Leveson

(2004). A new accident model for engineering safer systems. Safety Science, 42(4), 237–270. https://doi.org/10.1016/s0925-7535(03)00047-x

16.

Madhavan

Wiegmann

D. A.

(2007). Similarities and differences between human–human and human–automation trust: An integrative review. Theoretical Issues in Ergonomics Science, 8(4), 277–301. https://doi.org/10.1080/14639220500337708

17.

Medical Device

Interoperability.

(2023, May 16). U.S. food and drug administration. https://www.fda.gov/medical-devices/digital-health-center-excellence/medical-device-interoperability

18.

Moray

(2003) Monitoring, complacency, scepticism and eutactic behaviour. International Journal of Industrial Ergonomics, 31(3), 175–178. https://doi.org/10.1016/S0169-8141(02)00194-4

19.

Parasuraman

Riley

(1997). Humans and automation: Use, misuse, disuse, abuse. Human Factors: The Journal of the Human Factors and Ergonomics Society, 39(2), 230–253. https://doi.org/10.1518/001872097778543886

20.

Sheridan

Thomas B.

(1975). Considerations in modeling the human supervisory controller. IFAC Proceedings Volumes, 8(1, Part 3), 223–228. https://doi.org/10.1016/S1474-6670(17)67555-4

21.

Sheridan

T. B.

(2002). Chapter 1: Introduction. In Humans and automation: System design and research issues (pp. 3–13). John Wiley and Sons, Inc.

22.

Sheridan

T. B.

(2018). Comments on “Issues in human-automation interaction modeling: Presumptive Aspects of frameworks of types and levels of automation” by David B. Kaber. Journal of Cognitive Engineering and Decision Making, 12(1), 25–28. https://doi.org/10.1177/1555343417724964

23.

Skraaning

Jr. Jamieson

G. A.

(2023). The failure to grasp automation failure. Journal of Cognitive Engineering and Decision Making, Advance Online Copy. https://doi.org/10.1177/15553434231189375

24.

‘Inadequate safety culture’ contributed to uber automated test vehicle crash—NTSB calls for federal review process for automated vehicle testing on public roads. (2019, November 19). National Transportation Safety Board. https://www.ntsb.gov/news/press-releases/Pages/NR20191119c.aspx