The Effect of AI Benevolence on Humans’ Trust and Use of an AI Assistant in a Video Game

Abstract

Recent studies stress the importance of trust in artificial intelligence (AI) adoption. Previous studies have primarily focused on how the ability and integrity (transparency) of AI affect people’s trust and usage intention. However, few studies have explored the role of benevolence. This study investigates how the perceived benevolence of AI assistants influences users’ trust and willingness to interact with them. Through an interactive video game, participants interacted with either a benevolent AI assistant (prioritizing user safety and well-being) or a non-benevolent AI assistant (prioritizing its own goals). The AI assistant communicated with the subjects by using text boxes. After completing the task, participants completed a questionnaire that included topics about trust, perceived benevolence, and willingness to use. The results from two experiments showed that the benevolent AI assistant significantly enhanced perceived trust and increased users’ willingness to cooperate with the AI assistant in future interactions.

Introduction

Artificial intelligence (AI) permeates people’s daily activities, such as using digitization assistants like Alexa and Siri. AI is transforming multiple sectors, including transportation, healthcare, and education, significantly enhancing the quality of life (Silver et al., 2016). Despite these advancements, a critical issue remains. Many individuals hesitate to fully trust AI technologies (Lyons et al., 2024), raising a fundamental question: to what extent can AI be trusted? Research suggests that a lack of trust in AI can hinder its adoption and reduce the efficiency of its use (Glikson & Woolley, 2020). Given AI’s increasing role in decision-making processes, fostering public trust is essential for maximizing its societal benefits. To address the problem of public trust in AI, this study examined how the benevolence level of AI affected users’ trust and willingness to use.

Before discussing human trust in technology, it’s important to define trust. Mayer et al. (1995) defined trust between humans as the willingness to accept vulnerability, which is affected by credibility and trust stability inherently rooted in at least one quality of a person: ability, integrity, and benevolence. Furthermore, according to Lee and See (2004), trust can be viewed as the attitude of an agent (or artificial machine) to help achieve personal goals in uncertain and vulnerable situations. Building on these notions of reliability, Lee and See (2004) concluded that, for people to trust AI, this AI must also possess the above-mentioned attributes. As a result, researchers have begun to study the impact of ability and integrity on the human-machine trust process. For instance, Hancock et al. (2011) found a positive correlation between AI performance (ability) and trust. Additionally, Yu and Li (2022) emphasized that greater perceived transparency increases trust. However, limited research has explored the influence of the benevolence of AI.

Researchers have recently investigated the effect that benevolence may have on the human-machine trust process and the public’s willingness to use AI. Because of the limited literature on the benevolence of AI, studies have yielded varying results. In Schütz et al.’s (2023) study of robo-advisors, researchers found that benevolence did not significantly affect trust as anticipated. By contrast, Lyons et al. (2024) examined participants’ trust in autonomous security robots (ASRs) and found that benevolence did affect trust. A limitation of the previously mentioned contradictions using benevolent AI, Lyons et al.’s methodology only showed videos of autonomous security robot (ASR) behavior without actively engaging with the ASR. Also, the ASRs in the videos presented during the experiment did not exhibit emotional expression or employ social etiquette in their interactions with participants. Previous research fails to address indirect interaction, narrative behavior, lack of emotional expression in AI interactions, and reliance on self-reported surveys that pose challenges for accurately measuring trust in AI. Similarly, in Schütz et al.’s study (2023), the cues used to signal “high” versus “low” benevolence or ability were necessarily simplistic and text-based. The present study addresses these gaps by investigating the effect of AI benevolence on trust and willingness to use, using quantitative measures.

Approach

Using a between-subjects experimental design, this study used a novel interactive video game to examine the effects of AI benevolence on trust and willingness to use an AI assistant. In this game, the AI assistant appeared as a virtual robot. Furthermore, throughout the entire game task, the AI assistant communicated with the participants through the chat box. The language used by both the Benevolent and Non-benevolent AI assistants followed a consistent sentence structure to ensure comparability. Two experiments were conducted, and participants in Experiment 1 (N = 88) and Experiment 2 (N = 153) were randomly assigned to one of two conditions: (a) the Benevolent AI prioritized the participant’s well-being; or (b) the Non-benevolent AI prioritized its own success. The purpose of these experiments was to examine whether the benevolent AI assistant would be perceived as more trustworthy and increase willingness to use.

In Experiments 1 and 2, participants interacted with an AI assistant to complete a task (three rounds) that involved attacking enemies and obtaining the final key. In the first round, participants determined which path to choose to move forward and complete the task with the AI assistant. In the second round, participants joined the AI assistant in attacking a low-level monster enemy. In the third round, participants defeated the final boss with the AI assistant and obtained the key to success. In the video game, participants moved forward by clicking on the screen with the mouse and cooperated with the AI assistant to complete tasks and attack enemies. The game was programmed so that all participants completed the game successfully and participants could not die.

The AI’s behavior was manipulated in three areas: safety priorities, healing behavior, and final mission decisions. The AI’s communication was also manipulated to be Benevolent or Non-benevolent. In the first round, the Benevolent AI chose a safer but less efficient route, prioritizing safety over efficiency, while the Non-benevolent AI selected a more efficient but riskier path. In the second round, both the participant and the AI could be injured. Participants could not heal themselves; only the AI controlled healing. The AI’s choice to heal itself or the participant was made visually salient through an on-screen health bar, where green indicated good health and red indicated critical health. Although participants could not die, this feedback clearly revealed whether the AI prioritized the participant’s well-being or its own survival. In the final round, the Benevolent AI sacrificed its own health to help the participant obtain the key and succeed, whereas the Non-benevolent AI preserved its own health and forced the participant to face the challenge alone.

After completing the task, participants completed the Trustworthiness Scale and the Willingness to Use Scale to calculate trustworthiness and willingness to use (Ajzen & Fishbein, 1977; Lyons & Guznov, 2018; Mayer & Davis, 1995). In addition, participants were asked to complete the Perception of Benevolence Scale to demonstrate the effectiveness of the experiment in manipulating the AI’s benevolent or non-benevolent behavior. Participants also indicated through a binary question whether they would choose to cooperate with the AI assistant again in a future game. However, in Experiment 1, responses on one scale may have influenced others. Thus, to minimize the influence of previous ratings on subsequent responses, Experiment 2 removed the binary question and had participants rated only one of the three measures instead of all three. The order of the questionnaires was randomized but balanced across participants in Experiment 2.

Outcome

In Experiment 1, based on the chi-squared test revealed that there was a significant effect of AI benevolence on participants’ cooperation choices, indicating that participants in the Benevolent AI condition were statistically more likely to continue cooperating with the AI than those in the Non-Benevolent AI condition. Based on the independent-samples t-test, the overall score of Perception of Benevolence was rated significantly higher in the Benevolent AI condition compared to the Non-benevolent AI condition, indicating that the manipulation of AI behavior was successful. The overall score of Trustworthiness was also significantly higher in the Benevolent AI condition compared to the Non-benevolent AI condition. The overall score of Willingness to use was significantly greater in the Benevolent AI condition compared to the Non-benevolent AI condition. These findings indicate that AI benevolence fosters greater engagement with the AI and more positive user perceptions.

In Experiment 2, Perception of Benevolence was significantly higher in the Benevolent AI condition compared to the Non-benevolent AI condition. Trustworthiness was also significantly higher in the Benevolent AI condition than in the Non-benevolent AI condition. Willingness to Use AI was significantly greater in the Benevolent AI condition compared to the Non-benevolent AI condition. Thus, Experiment 2 results replicate Experiment 1.

Conclusion

The results of both experiments support the research hypothesis that benevolent AI can promote users’ trust and willingness to use. These findings suggest that when AI systems clearly prioritize user well-being through decisions based on user interests and safety, self-sacrificing healing behavior, and altruistic final task choices, users will respond more positively, in both cognitive and behavioral measures. The results further reinforce the idea that trust in AI is influenced not only by performance accuracy or efficiency but also by perceived intent and ethical alignment through perceived benevolence. Additionally, the findings show that the three manipulated dimensions—safety, healing behavior, and mission outcome—each shape users’ emotional and moral evaluations of AI assistants. This has important implications for the growing application of AI in fields such as healthcare and disaster response (Topol, 2019; Visave, 2024).

There are limitations to the study. This study was conducted in a controlled game environment, which may not fully reflect real-world AI interactions. Future research should explore real-world AI applications such as customer service bots, virtual assistants, and autonomous systems to assess whether these findings hold across various contexts.

In conclusion, the results contribute to human-computer interaction research, reinforcing the need for AI designers and developers to consider ethical and social dimensions beyond technical reliability. This study highlights that demonstrating the benevolent behavior of AI systems not only improves user experiences with AI products in everyday contexts, such as education, but also promotes progress and cooperation in human-AI interaction in high-risk fields, such as disaster response and healthcare.