The Importance of Studying and Guarding Against Systemic Automation Failures

Abstract

Skraaning and Jamieson (2023) remind us that automation failures are rarely the result of failures of individual components of the automated system itself but rather from unanticipated interactions within the larger system of which the automation is a part. Their preliminary taxonomy of automation failure mechanisms offers an important contribution to the literature. This commentary further expands the range of systemic automation failure mechanisms that researchers, designers, and evaluators need to be aware of and attempt to guard against—be it through analysis, design, or evaluation. The examples come from the Fukushima Nuclear Accident, automated vehicle accidents, as well as new failure mechanisms that arise from emerging machine learning (ML) technologies.

Keywords

automation failure taxonomy human automation interaction topics Fukushima systemic automated vehicles automation domains level of automation

Get full access to this article

View all access options for this article.

References

Bainbridge

(1983). Ironies of automation. Automatica, 19(6), 775–779. https://doi.org/10.1016/0005-1098(83)90046-8

Bote

(2023 August 15). Cruise vehicle gets stuck in wet concrete while driving in San Francisco. SFGATE. https://www.sfgate.com/tech/article/cruise-stuck-wet-concrete-sf-18297946.php

Caruana

Lou

Gehrke

Koch

Sturm

Elhadad

(2015). August). Intelligible models for healthcare: Predicting pneumonia risk and hospital 30-day readmission. In Proceedings of the 21th ACM SIGKDD international conference on knowledge discovery and data mining (pp. 1721–1730). Association for Computing Machinery.

Endsley

(2019). Human factors & aviation safety. Testimony to the United States house of representatives hearing on Boeing 737-Max 8 crashes. https://docs.house.gov/meetings/PW/PW00/20191211/110296/HHRG-116-PW00-Wstate-EndsleyM-20191211.pdf

Fowler

(2023 December 31). Testing Tesla’s autopilot recall, I don’t feel much safer – and neither should you. The Washington Post. https://www.washingtonpost.com/technology/2023/12/31/tesla-autopilot-recall-test/

Jamieson

G. A.

Skraaning

(2020). The absence of degree of automation trade-offs in complex work settings. Human Factors, 62(4), 516–529. https://doi.org/10.1177/0018720819842709

Lee

C. W.

Nayeer

Garcia

D. E.

Agrawal

Liu

(2020). Identifying the operational design domain for an automated driving system through assessed risk. 2020 IEEE intelligent vehicles symposium (IV), Las Vegas, NV, USA, 19 October, 2020–13 November, 2020, pp. 1317–1322. doi: 10.1109/IV47402.2020.9304552

Leveson

(2004). A new accident model for engineering safer systems. Safety Science, 42(4), 237–270. https://doi.org/10.1016/s0925-7535(03)00047-x

National Research Council . (2014). Lessons learned from the Fukushima nuclear accident for improving safety of U.S. Nuclear plants. The National Academies Press. https://doi.org/10.17226/18294

10.

Onnasch

Wickens

C. D.

Manzey

(2014). Human performance consequences of stages and levels of automation: An integrated meta-analysis. Human Factors, 56(3), 476–488. https://doi.org/10.1177/0018720813501549

11.

Perrow

(1984). Normal accidents: Living with high-risk technology. Basic Books, Inc.

12.

Rayo

M. F.

Horwood

C. R.

Fitzgerald

M. C.

Grayson

M. R.

Abdel-Rasoul

Moffatt-Bruce

S. D.

(2022). Situated visual alarm displays support machine fitness assessment for nonexplainable automation. IEEE Transactions on Human-Machine Systems, 52(5), 984–993. https://doi.org/10.1109/THMS.2022.3155714

13.

Roth

E. M.

DePass

Harter

Scott

Wampler

(2018). Beyond levels of automation: Developing more detailed guidance for human automation interaction design. Proceedings of the Human Factors and Ergonomics Society - Annual Meeting, 62(1), 150–154. https://doi.org/10.1177/1541931218621035

14.

Roth

E. M.

DePass

Scott

Truxler

Smith

Wampler

(2017). Designing collaborative planning systems: Putting joint cognitive systems principles to practice. In Smith

P. J.

Hoffman

R. R.

(Eds.), Cognitive systems engineering: The future for a changing world (pp. 247–268). Taylor & Francis, CRC Press.

15.

Sarter

N. B.

Mumaw

R. J.

Wickens

C. D.

(2007). Pilots’ monitoring strategies and performance on automated flight decks: An empirical study combining behavioral and eye-tracking data. Human Factors, 49(3), 347–357. https://doi.org/10.1518/001872007X196685

16.

Sarter

N. B.

Woods

D. D.

Billings

C. E.

(1997). Automation surprises. In Salvendy

(Ed.), Handbook of human factors (pp. 1026–1043). John Wiley & Sons.

17.

Schwartz

Vassilev

Greene

Perine

Burt

Hall

(2022). Towards a standard for identifying and managing bias in artificial intelligence. NIST Special Publication 1270. https://doi.org/10.6028/NIST.SP.1270.

18.

Siddiqui

Sands

Gregg

Thadani

Lerman

(2023, December 13). Tesla conducts largest-ever recall for ‘insufficient’ safety controls after exclusive post report on autopilot. The Washington Post. https://www.washingtonpost.com/technology/2023/12/13/tesla-autopilot-recall/

19.

Skraaning

Jr. Jamieson

G. A.

(2023). The failure to grasp automation failure. Journal of Cognitive Engineering and Decision Making, 0(0), 15553434231189375. https://doi.org/10.1177/15553434231189375.

20.

Testla . (nd). Update vehicle firmware to prevent driver Misuse of autosteer. Retrieved January 14, 2024 from. https://www.tesla.com/support/vehicle-firmware-prevent-autosteer-misuse

21.

Thadani

(2023, October 24). California suspends cruise’s autonomous driverless vehicle permits. The Washington Post. https://www.washingtonpost.com/technology/2023/10/24/cruise-robotaxis-california-suspended/

22.

Thadani

Siddiqui

Lerman

Merrill

(2023, December 10). Tesla drivers run autopilot where it’s not intended – with deadly consequences. The Washington Post. https://www.washingtonpost.com/technology/2023/12/10/tesla-autopilot-crash/

23.

U. S. NRC . (nd). Design-basis accident. Retrieved January 14, 2023. https://www.nrc.gov/reading-rm/basic-ref/glossary/design-basis-accident.html

24.

Weld

D. S.

Bansal

(2019). The challenge of crafting intelligible intelligence. Communications of the ACM, 62(6), 70–79. https://doi.org/10.1145/3282486

25.

Westin

Borst

Kampen

E. J.

Nunes

T. M.

Boonsong

Hilburn

Cocchioni

Bonelli

(2022). Personalized and transparent AI support for ATC conflict detection and resolution: An empirical study. Proceedings of the 12th SESAR Innovation Days, Budapest, Hungary, December 5-8, 2022, pp. 5–8.

26.

Wickens

(2018). Automation stages & levels, 20 years after. Journal of Cognitive Engineering and Decision Making, 12(1), 35–41. https://doi.org/10.1177/1555343417727438

27.

Wickens

C. D.

(2020). 21. Automation lessons from other domains. In Fisher

D. L.

Horrey

W. J.

Lee

J. D.

Regan

M. A.

(Eds.), Handbook of human factors for automated, connected, and intelligent vehicles (pp. 455–472). CRC Press.

28.

Wickens

C. D.

Santamaria

Sebok

Sarter

N. B.

(2010). Stages and levels of automation: An integrated meta-analysis. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 54(4), 389–393. https://doi.org/10.1177/154193121005400425