Abstract – The objective of this panel was to discuss issues related to the development and use of autonomous systems, with specific focus on the overriding themes of ethical considerations and potential liability for Human Factors and Ergonomics (HF/E) professionals who are involved in their development. Chris Brill provided opening remarks to frame the discussion and introduce the panelists. James Bliss discussed legal implications related to our collective penchant for developing conservative, false-alarm prone automation. Peter Hancock advocated for human-centered constraints on autonomous systems, as they may, one day, pose an existential threat to humanity. Dietrich Manzey discussed ethical considerations for autonomous systems, including how design can encourage ethical user behavior. Joachim Meyer argued that HF/E professionals have an obligation to help designers understand the ethical implications of poor design, particularly in the context of autonomous systems. Lastly, Alison Vredenburgh provided thoughts on potential liability for HF/E professionals, particularly in light of the relative newness of autonomous systems. The panel then turned to facilitated discussion with panelists and audience members. Specific themes included the boundaries of our responsibilities as HF/E professionals for ill-conceived or morally-objectionable systems, potential implications of manipulating user trust through design, cross-cultural perspectives on public acceptance and legal peril, and how concerns might differ by domain (e.g., medical vs. combat vs. manufacturing). The session concluded with panelists summarizing how ethics influence design and recommendations for how HF/E professionals can potentially protect themselves from legal liability for mishaps involving autonomous systems they helped develop.
BohmeS.R.EgilmanD. (2006).Consider the source: Warnings and anti- warnings in the tobacco, automobile, beryllium and pharmaceutical industries. In WogalterM. (Ed.), The Handbook of Warnings. (pp. 635-667). Mahwah, NJ: Lawrence Erlbaum Associates, Inc.
4.
BreznitzS. (1984): Cry wolf:The psychology of false alarms. Hillsdale, NJ: Lawrence Erlbaum Associates.
GraybielA.KneptonJ. (1976). Sopite syndrome: a sometimes sole manifestation of motion sickness. Aviation, Space, and Environmental Medicine, 47(8), 873-882.
10.
GettyD.J.SwetsJ.A.PickettR.M.GonthierD. (1995). System operator response to warnings of danger: A laboratory investigation of the effects of the predictive value of a warning on human response time. Journal of Experimental Psychology: Applied, 1(1), 19-33.
11.
HancockP.A. (2016). Imposing limits on autonomous systems. Manuscript in preparation.
LawsonB. D.MeadA. M. (1998). The sopite syndrome revisited: drowsiness and mood changes during real or apparent motion. Acta astronautica, 43(3), 181-192
SarterN. B.WoodsD. D.BillingsC. E. (1997). Automation surprises. In SalvendyG. (ed.), Handbook of human factors and ergonomics. 2nd ed. John Wiley and Sons.
19.
VredenburghA.G.ZackowitzI.B. (2006). Expectations. In WogalterM. (Ed.), The Handbook of Warnings. (pp. 345-354). Mahwah, NJ: Lawrence Erlbaum Associates, Inc.
20.
WeinerE.L. (1989). Human factors of advanced technology (“glass cockpit”) transport aircraft. (NASA Contractor Report No. 177528). NASA Ames Research Center.
