Determining the efficacy of two trust repair strategies (apology and denial) for trust violations of an ethical nature by an autonomous teammate.
Background
While ethics in human-AI interaction is extensively studied, little research has investigated how decisions with ethical implications impact trust and performance within human-AI teams and their subsequent repair.
Method
Forty teams of two participants and one autonomous teammate completed three team missions within a synthetic task environment. The autonomous teammate made an ethical or unethical action during each mission, followed by an apology or denial. Measures of individual team trust, autonomous teammate trust, human teammate trust, perceived autonomous teammate ethicality, and team performance were taken.
Results
Teams with unethical autonomous teammates had significantly lower trust in the team and trust in the autonomous teammate. Unethical autonomous teammates were also perceived as substantially more unethical. Neither trust repair strategy effectively restored trust after an ethical violation, and autonomous teammate ethicality was not related to the team score, but unethical autonomous teammates did have shorter times.
Conclusion
Ethical violations significantly harm trust in the overall team and autonomous teammate but do not negatively impact team score. However, current trust repair strategies like apologies and denials appear ineffective in restoring trust after this type of violation.
Application
This research highlights the need to develop trust repair strategies specific to human-AI teams and trust violations of an ethical nature.
AliS.PayneB. H.WilliamsR.ParkH. W.BreazealC. (2019). Constructionism, ethics, and creativity: Developing primary and middle school artificial intelligence education. International Workshop on Education in Artificial Intelligence K-12(EDUAI’19), 1-4.
ButlerJ. K.Jr.CantrellR. S. (1984). A behavioral decision theory approach to modeling dyadic trust in superiors and subordinates. Psychological Reports, 55(1), 19-28. https://doi.org/10.2466%2Fpr0.1984.55.1.19.
4.
CohenM. C.DemirM.ChiouE. K.CookeN. J. (2021). The dynamics of trust and verbal anthropomorphism in human-autonomy teaming. 2021. In: IEEE 2nd International Conference on Human-Machine Systems (ICHMS), 08–10 September 2021, Magdeburg, Germany, pp. 1–6
5.
CookeN. J.GormanJ.PedersenH.WinnerJ.DuranJ.TaylorA.AmazeenP. G.AndrewsD. H.RoweL. (2007). Acquisition and retention of team coordination in command-and-control. Cognitive Engineering Research INST Mesa AZ
6.
CookeN. J.ShopeS. M. (2017). Designing a synthetic task environment. In Scaled worlds: Development, validation and applications (pp. 273–288). Routledge
7.
CostaA. C.AndersonN. (2011). Measuring trust in teams: Development and validation of a multifaceted measure of formative and reflective indicators of team trust. European Journal of Work and Organizational Psychology, 20(1), 119-154. https://doi.org/10.1080/13594320903272083.
8.
De VisserE.ShawT.Mohamed-AmeenA.ParasuramanR. (2010). Modeling human-automation team performance in networked systems: Individual differences in working memory count. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 54(14), 1087–1091. https://doi.org/10.1177/154193121005401408
9.
DeLoneW.EspinosaJ. A.LeeG.CarmelE. (2005). Bridging global boundaries for IS project success. In: Proceedings of the 38th Annual Hawaii International Conference on System Sciences, 06 January 2005, Big Island, HI
10.
De VisserE.ParasuramanR. (2011). Adaptive aiding of human-robot teaming: Effects of imperfect automation on performance, trust, and workload. Journal of Cognitive Engineering and Decision Making, 5(2), 209–231. https://doi.org/10.1177/1555343411410160
11.
De VisserE. J.PakR.NeerincxM. A. (2017). Trust development and repair in human-robot teams. In: Proceedings of the companion of the 2017 ACM/IEEE international conference on human-robot interaction, TRADR: Long-Term Human-Robot Teaming for Disaster Response, pp. 103–104
12.
De VisserE. J.PakR.ShawT. H. (2018). From ‘automation’to ‘autonomy’: The importance of trust repair in human–machine interaction. Ergonomics, 61(1), 1–33. https://doi.org/10.1080/00140139.2018.1457725
13.
De VisserE. J.PeetersM. M.JungM. F.KohnS.ShawT. H.PakR.NeerincxM. A. (2020). Towards a theory of longitudinal trust calibration in human–robot teams. International Journal of Social Robotics, 12(2), 459–478. https://doi.org/10.1007/s12369-019-00596-x
FernandezR.ShahS.RosenmanE. D.KozlowskiS. W.ParkerS. H.GrandJ. A. (2017). Developing team cognition: A role for simulation. Simulation in Healthcare: Journal of the Society for Simulation in Healthcare, 12(2), 96–103. https://doi.org/10.1097/SIH.0000000000000200
16.
FlathmannC.SchelbleB. G.ZhangR.McNeeseN. J. (2021). Modeling and guiding the creation of ethical human-AI teams. In: Proceedings of the 2021 AAAI/ACM conference on AI (pp. 469–479). Ethics, and Society
17.
HoffKevinBashirMasooda (2015). Trust in Automation: Integrating Empirical Evidence on Factors That Influence Trust. Human Factors, 57(3), 407–434. https://doi.org/10.1177%2F0018720814547570
18.
JonesT. M.BowieN. E. (1998). Moral hazards on the road to the “virtual” corporation. Business Ethics Quarterly, 8(2), 273-292. https://doi.org/10.2307/3857329.
19.
Kasper-FuehreraE. C.AshkanasyN. M. (2001). Communicating trustworthiness and building trust in interorganizational virtual organizations. Journal of Management, 27(3), 235–254. https://doi.org/10.1177/014920630102700302
20.
KelleyJ. F. (2018). Wizard of Oz (WoZ) a yellow brick journey. Journal of Usability Studies, 13(3), 119–124
21.
KimP. H.FerrinD. L.CooperC. D.DirksK. T. (2004). Removing the shadow of suspicion: The effects of apology versus denial for repairing competence-versus integrity-based trust violations. Journal of Applied Psychology, 89(1), 104–118. https://doi.org/10.1037/0021-9010.89.1.104
22.
KoxE. S.KerstholtJ. H.HuetingT. F.de VriesP. W. (2021). Trust repair in human-agent teams: The effectiveness of explanations and expressing regret. Autonomous Agents and Multi-Agent Systems, 35(2), 1-20. https://doi.org/10.1007/s10458-021-09515-9.
LyonsJ. B.WynneK. T.MahoneyS.RoebkeM. A. (2019). Trust and human-machine teaming: A qualitative study. In Artificial intelligence for the internet of everything (pp. 101–116). Elsevier
26.
MachM.DolanS.TzafrirS. (2010). The differential effect of team members’ trust on team performance: The mediation role of team cohesion. Journal of Occupational and Organizational Psychology, 83(3), 771–794. https://doi.org/10.1348/096317909X473903
27.
ManzeyD.ReichenbachJ.OnnaschL. (2012). Human performance consequences of automated decision aids: The impact of degree of automation and system experience. Journal of Cognitive Engineering and Decision Making, 6(1), 57–87. https://doi.org/10.1177/1555343411433844
28.
MaulsbyD.Greenb ergS.ManderR. (1993). Prototyping an intelligent agent through wizard of Oz. In: Proceedings of the INTERACT’93 and CHI’93 conference on human factors in computing systems, ACM Press (pp. 277–284)
29.
MayerR. C.DavisJ. H.SchoormanF. D. (1995). An integrative model of organizational trust. Academy of Management Review, 20(3), 709–734. https://doi.org/10.2307/258792
30.
McNeeseN. J.DemirM.CookeN. J.MyersC. (2018). Teaming with a synthetic teammate: Insights into human-autonomy teaming. Human Factors, 60(2), 262–273. https://doi.org/10.1177/0018720817743223
31.
McNeeseN. J.DemirM.ChiouE. K.CookeN. J. (2021a). Trust and team performance in human–autonomy teaming. International Journal of Electronic Commerce, 25(1), 51–72. https://doi.org/10.1080/10864415.2021.1846854
32.
McNeeseN. J.DemirM.CookeN. J.SheM. (2021b). Team situation awareness and conflict: A study of human–machine teaming. Journal of Cognitive Engineering and Decision Making, 15(2-3), 83–96. https://doi.org/10.1177/15553434211017354
33.
O’NeillT.McNeeseN.BarronA.SchelbleB. (2022). Human–autonomy teaming: A Review and analysis of the empirical literature. Human Factors, 64(5), 904-938. https://doi.org/10.1177/0018720820960865.
34.
PakR.FinkN.PriceM.BassB.SturreL. (2012). Decision support aids with anthropomorphic characteristics influence trust and performance in younger and older adults. Ergonomics, 55(9), 1059–1072. https://doi.org/10.1080/00140139.2012.691554
35.
PakR.RoviraE.McLaughlinA. C.BaldwinN. (2017). Does the domain of technology impact user trust? Investigating trust in automation across different consumer-oriented domains in young adults, military, and older adults. Theoretical Issues in Ergonomics Science, 18(3), 199–220. https://doi.org/10.1080/1463922X.2016.1175523
36.
ParasuramanR.MillerC. A. (2004). Trust and etiquette in high-criticality automated systems. Communications of the ACM, 47(4), 51–55. https://doi.org/10.1145/975817.975844
37.
PatelL.ElliottA.StorlieE.KethireddyR.GoodmanK.DickeyW. (2020). Ethical and legal challenges during the COVID-19 pandemic—are we thinking about rural hospitals?The Journal of Rural Health, 37(1), 175-178. https://doi.org/10.1111/jrh.12447.
38.
QuinnD. B.PakR.De VisserE. J. (2017). Testing the efficacy of human-human trust repair strategies with machines. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 61(1), 1794–1798. https://doi.org/10.1177/1541931213601930
39.
RebenskyS.CarmodyK.FickeC.NguyenD.CarrollM.WildmanJ.ThayerA. (2021). Whoops! Something went wrong: Errors, trust, and trust repair strategies in human agent teaming. In: International Conference on Human-Computer Interaction, Springer International Publishing, pp. 95–106
40.
ReedG. S.PettyM. D.JonesN. J.MorrisA. W.BallengerJ. P.DelugachH. S. (2016). A principles-based model of ethical considerations in military decision making. The Journal of Defense Modeling and Simulation, 13(2), 195–211. https://doi.org/10.1177/1548512915581213
41.
ReevesB.NassC. (1996). The media equation: How people treat computers, television, and new media like real people. Cambridge University Press
42.
ReidenbachR. E.RobinD. P. (1988). Some initial steps toward improving the measurement of ethical evaluations of marketing activities. Journal of Business Ethics, 7(11), 871-879. https://doi.org/10.1007/BF00383050.
43.
ReidenbachR. E.RobinD. P. (1990). Toward the development of a multidimensional scale for improving evaluations of business ethics. Journal of Business Ethics, 9(8), 639–653. https://doi.org/10.1007/978-94-007-4126-3_3
44.
SchaeferK. E.ChenJ. Y.SzalmaJ. L.HancockP. A. (2016). A meta-analysis of factors influencing the development of trust in automation: Implications for understanding autonomy in future systems. Human Factors, 58(3), 377–400. https://doi.org/10.1177/0018720816634228
45.
SchweitzerM. E.HersheyJ. C.BradlowE. T. (2006). Promises and lies: Restoring violated trust. Organizational Behavior and Human Decision Processes, 101(1), 1–19. https://doi.org/10.1016/j.obhdp.2006.05.005
46.
SeboS. S.KrishnamurthiP.ScassellatiB. (2019) I don’t believe you”: Investigating the effects of robot trust violation and repair. In: 2019 14th ACM, IEEE International Conference on Human-Robot Interaction (HRI), pp. 57–65
47.
SuttonG. W.WashburnD. M.ComtoisL. L.MoeckelA. R. (2006). Professional ethics violations gender, forgiveness, and the attitudes of social work students. Journal of College and Character, 7(1), 1–7. https://doi.org/10.2202/1940-1639.1501
48.
TabachnickB. G.FidellL. S.UllmanJ. B. (2007) Using multivariate statistics (5). Pearson Boston
49.
TextorC.ZhangR.LopezJ.SchelbleB. G.McNeeseN. J.FreemanG.PakR.TossellC.De VisserE. J. (2022). Exploring the Relationship Between Ethics and Trust in Human-AI Teaming: A Mixed Methods Approach. Journal of Cognitive Engineering and Decision Making. https://doi.org/10.1177%2F15553434221113964.
50.
ThompsonM. M.ThompsonM. H.AdamsB. D. (2008). Moral and ethical dilemmas in Canadian forces military operation: Qualitative and descriptive analyses of commanders’ operational experiences. Defence Research and Development Toronto. https://apps.dtic.mil/sti/citations/ADA505336
51.
WinfieldA.JirotkaM. (2018). Ethical governance is essential to building trust in robotics and artificial intelligence systems. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, 376(2133), 1-13. https://doi.org/10.1098/rsta.2018.0085.
52.
ZhangR.McNeeseN. J.FreemanG.MusickG. (2021). An ideal human” expectations of AI teammates in human-AI teaming. Proceedings of the ACM on Human-Computer Interaction, 4(CSCW3), 1–25. https://doi.org/10.1145/3432945
