Wrong,Strong,and Silent: What Happens when Automated Systems With High Autonomy and High Authority Misbehave?

Abstract

Warnings about the risks of literal-minded automation—a system that can’t tell if its model of the world is the world it is actually in—have been sounded for over 70 years. The risk is that a system will do the “right” thing—its actions are appropriate given its model of the world, but it is actually in a different world—producing unexpected/unintended behavior and potentially harmful effects. This risk—wrong, strong, and silent automation—looms larger today as our ability to deploy increasingly autonomous systems and delegate greater authority to such systems expands. It already produces incidents, outages of valued services, financial losses, and fatal accidents across different settings. This paper explores this general and out-of-control risk by examining a pair of fatal aviation accidents which revolved around wrong, strong and silent automation.

Keywords

autonomy automation accidents risks of autonomy aviation safety aviation accidents supervisory control

Get full access to this article

View all access options for this article.

References

Dekker

S. W. A.

Layson

M. D.

Woods

D. D.

(2022). Repentance as rebuke: Betrayal and moral injury in safety engineering. Science and Engineering Ethics, 28(6), 56. https://doi.org/10.1007/s11948-022-00412-2

Dutch Safety Board (2002). Turkish airlines, crashed during approach, Boeing 737-800, TC-JGE Amsterdam Schiphol Airport. In Board

D. S.

(Ed.), Dutch Safety Board.

Eraslan

Yildiz

Annaswamy

A. M.

(2020). Shared control between pilots and autopilots: An illustration of a cyberphysical human system. IEEE Control Systems, 40(6), 77–97. https://doi.org/10.1109/MCS.2020.3019721

Farjadian

A. B.

Thomsen

Annaswamy

A. M.

Woods

D. D.

(2021). Resilient flight control: An architecture for human supervision of automation. IEEE Transactions on Control Systems Technology, 29(1), 29–42. https://doi.org/10.1109/TCST.2019.2959542

Johnson

Bradshaw

J. M.

Feltovich

P. J.

Jonker

C. M.

van Riemsdijk

M. B.

Sierhuis

(2014). Coactive design: Designing support for interdependence in joint activity. Journal of Human-Robot Interaction, 3(1), 43–69. https://doi.org/10.5898/jhri.3.1.johnson

Johnson

Bradshaw

J. M.

Feltovich

P. J.

(2017). Tomorrow’s human–machine design tools: From levels of automation to interdependencies. Journal of Cognitive Engineering and Decision Making, 12(1), 77–82. https://doi.org/10.1177/1555343417736462

Maguire

(2024a). Automation doesn’t work the way we think it does. IEEE Software, 41(1), 138–141. https://doi.org/10.1109/MS.2023.3328746

Maguire

(2024b). Joint cognitive systems: Ideas with impact for designing safer, more resilient automated systems. IEEE Software, 41(3), in press.

Morey

D. A.

Marquisee

J. M.

Gifford

R. C.

Fitzgerald

M. C.

Rayo

M. F.

(2020). Predicting graceful extensibility of human-machine systems: A new analysis method for evaluating extensibility plots to anticipate distributed system performance. Proceedings of the human factors and ergonomics society - annual meeting (Vol. 64, No. 1, pp. 313–318). Sage Publications.

10.

Nakahira

Liu

Sejnowski

T. J.

Doyle

J. C.

(2021). Diversity-enabled sweet spots in layered architectures and speed–accuracy trade-offs in sensorimotor control. Proceedings of the National Academy of Sciences of the United States of America, 118, Article e1916367118. https://doi.org/10.1073/pnas.1916367118

11.

Sarter

N. B.

(2002). Multimodal information presentation in support of human-automation communication and coordination. In Salas

(Ed.), Advances in human performance and cognitive engineering research: Automation (pp. 13–35). Elsevier Science/JAI Press. https://doi.org/10.1016/S1479-3601(02)02004-0

12.

Sarter

N. B.

Woods

D. D.

(1995). “How in the world did we get into that mode?” Mode error and awareness in supervisory control. Human Factors: The Journal of the Human Factors and Ergonomics Society, 37(1), 5–19. https://doi.org/10.1518/001872095779049516

13.

Sarter

N. B.

Woods

D. D.

(1997). Team play with a powerful and independent agent: Operational experiences and automation surprises on the airbus A-320. Human Factors, 39(4), 553–569. https://doi.org/10.1518/001872097778667997

14.

Schraagen

J.-M.

Barnhoorn

J. S.

van Schendel

van Vught

(2022). Supporting teamwork in hybrid multi-team systems. Theoretical Issues in Ergonomics Science, 23(2), 199–220. https://doi.org/10.1080/1463922X.2021.1936277

15.

Wiener

(1950). The human use of human beings: Cybernetics and society. Doubleday.

16.

Woods

D. D.

Sarter

N. B.

(2000). Learning from automation surprises and going sour accidents. In Sarter

Amalberti

(Eds.), Cognitive engineering in the aviation domain (pp. 327–354). Erlbaum.

17.

Woods

D. D.

Patterson

E. S.

(2001). How unexpected events produce an escalation of cognitive and coordinative demands. PA Hancock, & PA Desmond, stress, workload, and fatigue. L. Erlbaum.

18.

Woods

D. D.

Hollnagel

(2006). Joint cognitive systems: Patterns in cognitive systems engineering. Taylor & Francis.

19.

Woods

D. D.

Dekker

S.W.A.

Cook

R.I.

Johannesen

L. L.

Sarter

N. B.

. (2010). Behind human error (2nd Edition). Ashgate.

20.

Woods

D. D.

(2016). The risks of autonomy: Doyle’s catch. Journal of Cognitive Engineering and Decision Making, 10(2), 131–133. https://doi.org/10.1177/1555343416653562

21.

Woods

D. D.

(2018). The theory of graceful extensibility: Basic rules that govern adaptive systems. Environment Systems and Decisions, 38(4), 433–457. https://doi.org/10.1007/s10669-018-9708-3

22.

Woods

D. D.

Balkin

E. A.

(2018). A resiliency trade space study of detect and avoid autonomy on drones when communications are degraded. Report for NASA Ames Research Center.

23.

Woods

D. D.

(2019). Cover story: Misbehaving automation. Air Traffic Management Magazine, 2019(2), 22–25.

24.

Woods

D. D.

(2021). How to kill zombie ideas: Why do people tenaciously believe myths about the relationship between people & technology? Invited talk. Georgia Institute of Technology. https://repository.gatech.edu/entities/publication/f362ff01-a8b3-4734-89ca-95d0b2c1f647

25.

Woods

D. D.

(2024). Limits of automata—then and now: Challenges of architecture, brittleness, and scale. Journal of Cognitive Engineering and Decision Making, 10(2), 131–133. https://doi.org/10.1177/1555343416653562

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.11 MB