Exploring the Relationship Between Ethics and Trust in Human–Artificial Intelligence Teaming: A Mixed Methods Approach

Abstract

Advancements and implementations of autonomous systems coincide with an increased concern for the ethical implications resulting from their use. This is increasingly relevant as autonomy fulfills teammate roles in contexts that demand ethical considerations. As AI teammates (ATs) enter these roles, research is needed to explore how an AT’s ethics influences human trust. This current research presents two studies which explore how an AT’s ethical or unethical behavior impacts trust in that teammate. In Study 1, participants responded to scenarios of an AT recommending actions which violated or abided by a set of ethical principles. The results suggest that ethicality perceptions and trust are influenced by ethical violations, but only ethicality depends on the type of ethical violation. Participants in Study 2 completed a focus group interview after performing a team task with a simulated AT that committed ethical violations and attempted to repair trust (apology or denial). The focus group responses suggest that ethical violations worsened perceptions of the AT and decreased trust, but it could still be trusted to perform tasks. The AT’s apologies and denials did not repair damaged trust. The studies’ findings suggest a nuanced relationship between trust and ethics and a need for further investigation into trust repair strategies following ethical violations.

Keywords

ethical artificial intelligence human–artificial intelligence teaming trust repair trust

Get full access to this article

View all access options for this article.

References

Abdi

(2010). The greenhouse-geisser correction. Encyclopedia of Research Design, 1(1), 544–548. http://dx.doi.org/10.4135/9781412961288.n168

Andres

H. P.

(2012). Technology-mediated collaboration, shared mental model and task performance. Journal of Organizational and End User Computing (JOEUC), 24(1), 64–81. https://doi.org/10.4018/joeuc.2012010104

Arrieta

A. B.

Díaz-Rodríguez

Del Ser

Bennetot

Tabik

Barbado

Garcia

Gil-Lopez

Molina

Benjamins

Chatila

Herrera

(2020). Explainable artificial intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI. Information Fusion, 58, 82–115. https://doi.org/10.1016/j.inffus.2019.12.012

Barnett

Bass

Brown

(1994). Ethical ideology and ethical judgment regarding ethical issues in business. Journal of Business Ethics, 13(6), 469–480. https://doi.org/10.1007/bf00881456

Bergman

Fassihi

(2021, September 18). The scientist and the A.I.-assisted, remote-control killing machine. The New York Times. https://www.nytimes.com/2021/09/18/world/middleeast/iran-nuclear-fakhrizadeh-assassination-israel.html

Bohemia Interactive [Video game] (2013). ArmA 3. : Marek Španěl.

Braun

Clarke

(2012). Thematic analysis. In Cooper

Camic

P. M.

Long

D. L.

, et al. (Eds.), APA handbook of research methods in psychology, vol. 2: Research designs: Quantitative, qualitative, neuropsychological, and biological (pp. 57–71). American Psychological Association.

Brundage

Avin

Clark

Toner

Eckersley

Garfinkel

Dafoe

Scharre

Zeitzoff

Filar

(2018). The malicious use of artificial intelligence: Forecasting, prevention, and mitigation. ArXiv Preprint ArXiv:1802.07228.

Cadsby

C. B.

Song

(2016). In-group favoritism and moral decision-making. Journal of Economic Behavior & Organization, 128, 59–71. https://doi.org/10.1016/j.jebo.2016.05.008

10.

Chen

J. Y.

Lakhmani

S. G.

Stowers

Selkowitz

A. R.

Wright

J. L.

Barnes

(2018). Situation awareness-based agent transparency and human-autonomy teaming effectiveness. Theoretical Issues in Ergonomics Science, 19(3), 259–282. https://doi.org/10.1080/1463922x.2017.1315750

11.

Clancy

(2019). Breakdowns in human-Ai partnership: Revelatory cases of automation bias in autonomous vehicle accidents. https://cdr.lib.unc.edu/concern/masters_papers/d791sm69k

12.

Cohen

M. C.

Demir

Chiou

E. K.

Cooke

N. J.

(2021, September). The dynamics of trust and verbal anthropomorphism in human-autonomy teaming. In 2021 IEEE 2nd International Conference on Human-Machine Systems (ICHMS), Magdeburg, Germany, 08-10 September 2021 (pp. 1–6). IEEE.

13.

Cointe

Bonnet

Boissier

(2016). Ethical judgment of agents’ behaviors in multi-agent systems. Proceedings of the 2016 International Conference on Autonomous Agents & Multiagent Systems, Singapore, May 9-May 13 (pp. 1106–1114).

14.

Cooke

N. J.

Salas

Cannon-Bowers

J. A.

Stout

R. J.

(2000). Measuring team knowledge. Human Factors, 42(1), 151–173. https://doi.org/10.1518/001872000779656561

15.

Craft

J. L.

(2013). A review of the empirical ethical decision-making literature: 2004–2011. Journal of Business Ethics, 117(2), 221–259. https://doi.org/10.1007/s10551-012-1518-9

16.

Cummings

Huang

Zhu

Finkelstein

Wei

(2019). The impact of increasing autonomy on training requirements in a UAV supervisory control task. Journal of Cognitive Engineering and Decision-Making, 13(4), 295–309. https://doi.org/10.1177/1555343419868917

17.

Demir

McNeese

N. J.

Cooke

N. J.

(2017). Team situation awareness within the context of human-autonomy teaming. Cognitive Systems Research, 46, 3–12. https://doi.org/10.1016/j.cogsys.2016.11.003

18.

Demir

McNeese

N. J.

Gorman

J. C.

Cooke

N. J.

Myers

C. W.

Grimm

D. A.

(2021). Exploration of teammate trust and interaction dynamics in human-autonomy teaming. IEEE Transactions on Human-Machine Systems, 51(6), 696–705. https://doi.org/10.1109/thms.2021.3115058

19.

Desai

Stubbs

Steinfeld

Yanco

(2009). Creating trustworthy robots: Lessons and inspirations from automated systems. In Proceedings of the AISB Convention: New Frontiers in Human-Robot Interaction, Edinburgh, Scotland, April 6 – April 9.

20.

de Visser

Parasuraman

(2011). Adaptive aiding of human-robot teaming: Effects of imperfect automation on performance, trust, and workload. Journal of Cognitive Engineering and Decision-Making, 5(2), 209–231. https://doi.org/10.1177/1555343411410160

21.

de Visser

E. J.

Monfort

S. S.

McKendrick

Smith

M. A.

McKnight

P. E.

Krueger

Parasuraman

(2016). Almost human: Anthropomorphism increases trust resilience in cognitive agents. Journal of Experimental Psychology: Applied, 22(3), 331–349. https://doi.org/10.1037/xap0000092

22.

de Visser

E. J.

Pak

Neerincx

M. A.

(2017). Trust development and repair in human-robot teams. Proceedings of the Companion of the 2017 ACM/IEEE International Conference on Human-Robot Interaction, Vienna, Austria, March 6 ‐ March 9 (pp. 103–104).

23.

de Visser

E. J.

Pak

Shaw

T. H.

(2018). From ‘automation’ to ‘autonomy’: The importance of trust repair in human–machine interaction. Ergonomics, 61(10), 1409–1427. https://doi.org/10.1080/00140139.2018.1457725

24.

de Visser

E. J.

Peeters

M. M.

Jung

M. F.

Kohn

Shaw

T. H.

Pak

Neerincx

M. A.

(2020). Towards a theory of longitudinal trust calibration in human–robot teams. International Journal of Social Robotics, 12(2), 459–478. https://doi.org/10.1007/s12369-019-00596-x

25.

de Vries

Midden

Bouwhuis

(2003). The effects of errors on system trust, self-confidence, and the allocation of control in route planning. International Journal of Human-Computer Studies, 58(6), 719–735. https://doi.org/10.1016/s1071-5819(03)00039-9

26.

Doran

Schulz

Besold

T. R.

(2017). What does explainable AI really mean? A new conceptualization of perspectives. arXiv preprint arXiv:1710.00794.

27.

Endsley

M. R.

(2017). From here to autonomy: Lessons learned from human–automation research. Human Factors, 59(1), 5–27. https://doi.org/10.1177/0018720816681350

28.

Esterwood

Robert

(2022). Having the right attitude: How attitude impacts trust repair in human-robot interaction. In 2022 ACM/IEEE International Conference on Human-Robot Interaction, At: Online.

29.

Feldman

J. A.

Sproull

R. F.

(1977). Decision theory and artificial intelligence II: The hungry monkey. Cognitive Science, 1(2), 158–192. https://doi.org/10.1207/s15516709cog0102_2

30.

Fitts

P. M.

(Ed.) (1951). Human engineering for an effective air-navigation and traffic-control system. National Research Council.

31.

Flathmann

Schelble

B. G.

Zhang

McNeese

N. J.

(2021). Modeling and guiding the creation of ethical human-AI teams. In Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, Virtual, May 19 – May 21 (pp. 469–479).

32.

Gunning

Aha

(2019). DARPA’s explainable artificial intelligence (XAI) program. AI Magazine, 40(2), 44–58. https://doi.org/10.1609/aimag.v40i2.2850

33.

Hancock

P. A.

Billings

D. R.

Schaefer

K. E.

Chen

J. Y.

De Visser

E. J.

Parasuraman

(2011). A meta-analysis of factors affecting trust in human-robot interaction. Human Factors, 53(5), 517–527. https://doi.org/10.1177/0018720811417254

34.

Himmelreich

(2018). Never mind the trolley: The ethics of autonomous vehicles in mundane situations. Ethical Theory and Moral Practice: An International Forum, 21(3), 669–684. https://doi.org/10.1007/s10677-018-9896-4

35.

Hoff

K. A.

Bashir

(2015). Trust in automation: Integrating empirical evidence on factors that influence trust. Human Factors, 57(3), 407–434. https://doi.org/10.1177/0018720814547570

36.

Hunt

S. D.

Vitell

(1986). A general theory of marketing ethics. Journal of Macromarketing, 6(1), 5–16. https://doi.org/10.1177/027614678600600103

37.

Hursthouse

(1999). On virtue ethics. OUP.

38.

Jobin

Ienca

Vayena

(2019). The global landscape of AI ethics guidelines. Nature Machine Intelligence, 1(9), 389–399. https://doi.org/10.1038/s42256-019-0088-2

39.

Jones

T. M.

Bowie

N. E.

(1998). Moral hazards on the road to the "virtual" corporation. Business Ethics Quarterly, 8(2), 273–292. https://doi.org/10.2307/3857329

40.

Kaber

D. B.

(2018). Issues in human–automation interaction modeling: Presumptive aspects of frameworks of types and levels of automation. Journal of Cognitive Engineering and Decision-Making, 12(1), 7–24. https://doi.org/10.1177/1555343417737203

41.

Kasper-Fuehrera

E. C.

Ashkanasy

N. M.

(2001). Communicating trustworthiness and building trust in interorganizational virtual organizations. Journal of Management, 27(3), 235–254. https://doi.org/10.1016/s0149-2063(01)00090-3

42.

Kelley

J. F.

(2018). Wizard of Oz (WoZ) a yellow brick journey. Journal of Usability Studies, 13(3), 119–124.

43.

Kim

P. H.

Ferrin

D. L.

Cooper

C. D.

Dirks

K. T.

(2004). Removing the shadow of suspicion: the effects of apology versus denial for repairing competence-versus integrity-based trust violations. Journal of Applied Psychology, 89(1), 104–118. https://doi.org/10.1037/0021-9010.89.1.104

44.

Kim

Song

(2021). How should intelligent agents apologize to restore trust? Interaction effects between anthropomorphism and apology attribution on trust repair. Telematics and Informatics, 61, 101595. https://doi.org/10.1016/j.tele.2021.101595

45.

Kohn

S. C.

Quinn

Pak

de Visser

E. J.

Shaw

T. H.

(2018, September). Trust repair strategies with self-driving vehicles: An exploratory study. In Proceedings of the Human Factors and Ergonomics Society Annual Meeting Philadelphia, PA, October 1 – October 5 (Vol. 62, No. 1, pp. 1108–1112). SAGE Publications.

46.

Kox

E. S.

Kerstholt

J. H.

Hueting

T. F.

De Vries

P. W.

(2021). Trust repair in human-agent teams: the effectiveness of explanations and expressing regret. Autonomous Agents and Multi-Agent Systems, 35(2), 1–20. https://doi.org/10.1007/s10458-021-09515-9

47.

Kuntz

J. R. C.

Kuntz

J. R.

Elenkov

Nabirukhina

(2013). Characterizing ethical cases: A cross-cultural investigation of individual differences, organisational climate, and leadership on ethical decision-making. Journal of Business Ethics, 113(2), 317–331. https://doi.org/10.1007/s10551-012-1306-6

48.

Lee

Moray

(1992). Trust, control strategies and allocation of function in human-machine systems. Ergonomics, 35(10), 1243–1270. https://doi.org/10.1080/00140139208967392

49.

Lee

J. D.

Moray

(1994). Trust, self-confidence, and operators’ adaptation to automation. International Journal of Human-Computer Studies, 40(1), 153–184. https://doi.org/10.1006/ijhc.1994.1007

50.

Lee

J. D.

See

K. A.

(2004). Trust in automation: Designing for appropriate reliance. Human Factors, 46(1), 50–80. https://doi.org/10.1518/hfes.46.1.50.30392

51.

Jiang

Yang

Wang

Fan

Liu

(2010, November). Development of an anthropomorphic prosthetic hand for man-machine interaction. In International Conference on Intelligent Robotics and Applications, Shanghai, China, November 10 ‐ November 12 (pp. 38–46). Springer.

52.

Luo

Huang

Peng

Song

Liu

(2021, August). Repairing human trust by promptly correcting robot mistakes with an attention transfer model. In 2021 IEEE 17th International Conference on Automation Science and Engineering (CASE), Lyon, France, August 23 – August 27 (pp. 1928–1933). IEEE.

53.

Lyons

J. B.

Wynne

K. T.

Mahoney

Roebke

M. A.

(2019). Trust and human-machine teaming: A qualitative study. In Artificial intelligence for the internet of everything (pp. 101–116). Academic Press.

54.

Majumdar Roy Choudhury

Aoun

Badawy

de Alburquerque

L. A.

Marjane

Wilkinson

(2021). Final report of the panel of experts on Libya established pursuant to security council resolution 1973 (2011) (S/2021/229). United Nations Security Council.

55.

Martinez-Martin

(2019). What are important ethical implications of using facial recognition technology in health care? AMA Journal of Ethics, 21(2), 180–187. https://doi.org/10.1001/amajethics.2019.180

56.

Maulsby

Greenberg

Mander

(1993, May). Prototyping an intelligent agent through Wizard of Oz. In Proceedings of the INTERACT'93 and CHI'93 conference on Human factors in computing systems (pp. 277–284).

57.

Maxmen

(2018). Self-driving car dilemmas reveal that moral choices are not universal. Nature, 562(7728), 469–469. https://doi.org/10.1038/d41586-018-07135-0

58.

Mayer

R. C.

Davis

J. H.

Schoorman

F. D.

(1995). An integrative model of organizational trust. Academy of Management Review, 20(3), 709–734. https://doi.org/10.5465/amr.1995.9508080335

59.

McNeese

N. J.

Demir

Chiou

E. K.

Cooke

N. J.

(2021a). Trust and team performance in human–autonomy teaming. International Journal of Electronic Commerce, 25(1), 51–72. https://doi.org/10.1080/10864415.2021.1846854

60.

McNeese

N. J.

Demir

Cooke

N. J.

Myers

(2018). Teaming with a synthetic teammate: Insights into human-autonomy teaming. Human Factors, 60(2), 262–273. https://doi.org/10.1177/0018720817743223

61.

McNeese

N. J.

Demir

Cooke

N. J.

She

(2021b). Team situation awareness and conflict: A study of human–machine teaming. Journal of Cognitive Engineering and Decision-Making, 15(2-3), 83–96. https://doi.org/10.1177/15553434211017354

62.

Mosier

K. L.

Skitka

L. J.

(1996). Human decision makers and automated decision aids: Made for each other? In Automation and human performance: Theory and applications (Vol. 514, pp. 201–220. Erlbaum.

63.

Muir

B. M.

(1987). Trust between humans and machines, and the design of decision aids. International Journal of Man-Machine Studies, 27(5-6), 527–539. https://doi.org/10.1016/s0020-7373(87)80013-5

64.

Nayyar

Wagner

A. R.

(2018, November). When should a robot apologize? understanding how timing affects human-robot trust repair. In International conference on social robotics, Cham, Qingdao, China, November 28 ‐ November 30 (pp. 265–274). Springer.

65.

O’Neill

McNeese

Barron

Schelble

(2020). Human–autonomy teaming: a review and analysis of the empirical literature. Human Factors. Advance online publication. https://doi.org/10.1177/0018720820960865

66.

Pak

Fink

Price

Bass

Sturre

(2012). Decision support aids with anthropomorphic characteristics influence trust and performance in younger and older adults. Ergonomics, 55(9), 1059–1072. https://doi.org/10.1080/00140139.2012.691554

67.

Parasuraman

Manzey

D. H.

(2010). Complacency and bias in human use of automation: An attentional integration. Human Factors, 52(3), 381–410. https://doi.org/10.1177/0018720810376055

68.

Parasuraman

Miller

C. A.

(2004). Trust and etiquette in high-criticality automated systems. Communications of the ACM, 47(4), 51–55. https://doi.org/10.1145/975817.975844

69.

Parasuraman

Riley

(1997). Humans and automation: Use, misuse, disuse, abuse. Human Factors, 39(2), 230–253. https://doi.org/10.1518/001872097778543886

70.

Parasuraman

Sheridan

T. B.

Wickens

C. D.

(2008). Situation awareness, mental workload, and trust in automation: Viable, empirically supported cognitive engineering constructs. Journal of Cognitive Engineering and Decision Making, 2(2), 140–160. https://doi.org/10.1518/155534308x284417

71.

Pistono

Yampolskiy

R. V.

(2016, July 9 ‐ XXX). Unethical research: How to create a malevolent artificial intelligence. Presented at the 25th International Joint Conference on Artificial Intelligence. Ethics for Artificial Intelligence Workshop, New York, NY.

72.

Quinn

D. B.

Pak

de Visser

E. J.

(2017). Testing the efficacy of human-human trust repair strategies with machines. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 61(1), 1794–1798. https://doi.org/10.1177/1541931213601930

73.

Rebensky

Carmody

Ficke

Nguyen

Carroll

Wildman

Thayer

(2021, July). Whoops! Something went wrong: Errors, trust, and trust repair strategies in human agent teaming. In International Conference on Human-Computer Interaction, Cham, Washington, DC, USA, July 24 - July 29 (pp. 95–106). Springer.

74.

Reed

G. S.

Petty

M. D.

Jones

N. J.

Morris

A. W.

Ballenger

J. P.

Delugach

H. S.

(2016). A principles-based model of ethical considerations in military decision-making. The Journal of Defense Modeling and Simulation, 13(2), 195–211. https://doi.org/10.1177/1548512915581213

75.

Rest

J. R.

(Ed.). (1994). Moral development in the professions: Psychology and applied ethics. Psychology Press.

76.

Robinette

Howard

A. M.

Wagner

A. R.

(2015, October). Timing is key for robot trust repair. In International conference on social robotics, Cham, Paris, France, October 26 - October 30 (pp. 574–583). Springer.

77.

Schaefer

K. E.

Chen

J. Y.

Szalma

J. L.

Hancock

P. A.

(2016). A meta-analysis of factors influencing the development of trust in automation: Implications for understanding autonomy in future systems. Human Factors, 58(3), 377–400. https://doi.org/10.1177/0018720816634228

78.

Schweitzer

M. E.

Hershey

J. C.

Bradlow

E. T.

(2006). Promises and lies: Restoring violated trust. Organizational Behavior and Human Decision Processes, 101(1), 1–19. https://doi.org/10.1016/j.obhdp.2006.05.005

79.

Sebo

S. S.

Krishnamurthi

Scassellati

(2019). “I don’t believe you”: Investigating the effects of robot trust violation and repair. In 2019 14th ACM/IEEE International Conference on Human-Robot Interaction (HRI), Daegu, Korea, March 11 - March 14 (pp. 57–65).

80.

Shneiderman

(2020). Bridging the gap between ethics and practice: Guidelines for reliable, safe, and trustworthy Human-Centered AI systems. ACM Transactions on Interactive Intelligent Systems (TiiS), 10(4), 1–31. https://doi.org/10.1145/3419764

81.

Sparks

J. R.

Pan

(2010). Ethical judgments in business ethics research: Definition, and research agenda. Journal of Business Ethics, 91(3), 405–418. https://doi.org/10.1007/s10551-009-0092-2

82.

Sutton

G. W.

Washburn

D. M.

Comtois

L. L.

Moeckel

A. R.

(2006). Professional ethics violations gender, forgiveness, and the attitudes of social work students. Journal of College and Character, 7(1), 1–7. https://doi.org/10.2202/1940-1639.1501

83.

Tashakkori

Creswell

J. W.

(2007). The new era of mixed methods. Journal of Mixed Methods Research, 1(1), 3–7. https://doi.org/10.1177/2345678906293042

84.

Walliser

J. C.

De Visser

E. J.

Shaw

T. H.

(2016). Application of a system-wide trust strategy when supervising multiple autonomous agents. In Proceedings of the Human Factors and Ergonomics Society Annual Meeting. 60(1), 133–137. https://doi.org/10.1177/1541931213601031

85.

Walliser

J. C.

de Visser

E. J.

Wiese

Shaw

T. H.

(2019). Team structure and team building improve human–machine teaming with autonomous agents. Journal of Cognitive Engineering and Decision Making, 13(4), 258–278. https://doi.org/10.1177/1555343419867563

86.

Wellman

M. P.

Doyle

(1992). Modular utility representation for decision-theoretic planning. Artificial Intelligence Planning Systems, 1992, 236–242. https://doi.org/10.1016/b978-0-08-049944-4.50033-1

87.

Zhang

McNeese

N. J.

Freeman

Musick

(2021). “An ideal human” Expectations of AI teammates in human-AI teaming. Proceedings of the ACM on Human-Computer Interaction, 4(CSCW3), 1–25. https://doi.org/10.1145/3432945

88.

Zhou

Chen

Berry

Reed

Zhang

Savage

(2020). A survey on ethical principles of AI and implementations. In 2020 IEEE Symposium Series on Computational Intelligence (SSCI), Canberra, Australia, December 1 ‐ December 4 (pp. 3010–3017).