Game On: Pioneering Human-Machine Teaming Research in Online Gaming Arenas

Abstract

This manuscript explores the use of online gaming environments to enhance human-machine teaming (HMT) research. Traditional HMT studies often struggle with participant recruitment, engagement, and ecological validity. Leveraging platforms like Minecraft, we discuss the integration of gamification to boost engagement and retention, simulate complex team dynamics, and expand recruitment reach. A case study within DARPA’s ASIST program demonstrated successful participant recruitment and data collection using competitive gaming elements and other incentives. The findings highlight the potential of online games for robust HMT research, suggesting future studies to explore scalability, transferability of results, and the balance of intrinsic and extrinsic motivators for optimal engagement.

Introduction

Human-machine teaming (HMT) research focuses on enhancing the cooperative dynamics between humans and artificial intelligence (AI) agents. The dynamic and interactive nature of teaming necessitates innovative methodological approaches to achieve ecological validity in HMT research. As such, online gaming environments have been explored as highly customizable platforms for observing the complexities of interactions in HMTs. Another challenge in HMT research that online games can help address is participant recruitment, which often requires significant time commitments from laboratory personnel, substantial physical space for team activities, and large budgets for participant incentives. However, traditional recruitment methods used in many psychological and social studies can still fall short in such gaming communities. In this paper, we describe various participant recruitment frameworks and strategies our team employed for a Minecraft-based experiment as part of the Defense Advanced Research Projects Agency (DARPA)’s Artificial Social Intelligence for Successful Teams (ASIST) program.

Leveraging Online Gaming Environments for HMT Research

The integration of online gaming environments into HMT research serves multiple objectives: boosting participant engagement and retention, simulating complex team dynamics, expanding recruitment reach, and exploring advanced HMT concepts. Online games are known for their immersive nature, which can enhance participant involvement and ensure prolonged engagement, leading to richer data collection. The interactive scenarios in online games closely mimic real-life challenges, thereby improving the ecological validity of the research. Furthermore, the global accessibility and appeal of online games allow researchers to attract a diverse participant pool, which is crucial for the robustness and generalizability of research findings (Gee, 2003; Williams et al., 2008).

Enhancing Participant Engagement and Retention

One of the primary challenges in HMT research is maintaining high levels of participant engagement and retention. Traditional recruitment methods often struggle to sustain participant engagement over extended periods, leading to incomplete data and reduced sample sizes.

The concept of gamification, which involves incorporating game-like elements into non-game contexts, has been shown to enhance engagement and motivation in various domains (Hamari et al., 2014). Gamification leverages the psychological principles of motivation, including intrinsic and extrinsic factors, to foster sustained engagement. Intrinsic motivation, driven by the inherent enjoyment of the activity, is a powerful force in gaming environments. Players often experience flow, a state of deep immersion and enjoyment, which can lead to extended periods of participation (Csikszentmihalyi, 1990). Extrinsic motivation, such as rewards and recognition, further enhances this engagement by providing tangible incentives for participation and performance. By combining these motivational strategies, researchers can create a compelling environment that keeps participants actively involved in the study.

Online gaming environments often naturally include gamification elements, enriching HMT research environments with the intrinsic motivational pull of gaming. For example, using leaderboards, rewards, and competitive events can motivate participants to engage more deeply with the research tasks and generate more robust data. These gamification features are often readily found in online gaming environments, enabling HMT researchers to create more seamlessly enjoyable and engaging experiences for participants, leading to higher retention rates and richer data collection.

The concept of flow, characterized by deep focus and enjoyment, is also closely related to engagement and cognitive performance (Csikszentmihalyi, 1990). Game-based testbeds are well-suited to facilitate flow by providing tasks that are appropriately challenging and match the skill levels of participants. When participants experience flow, they are more likely to remain engaged and perform tasks with greater efficiency and creativity. Flow enhances cognitive performance by promoting sustained attention and intrinsic motivation. This makes game-based testbeds an ideal platform for studying complex cognitive processes in a realistic and engaging manner. The interactive and challenging nature of games helps maintain participants’ interest and encourages them to fully engage with the tasks, leading to richer data and more accurate insights into cognitive processes.

Simulating Complex Team Dynamics

The ability to simulate complex team dynamics is another significant advantage of using online gaming environments for HMT research. Traditional laboratory-based experiments often struggle to replicate the multifaceted nature of real-world teaming scenarios. In contrast, online games like Roblox, Minecraft, and Overcooked! 2 can provide a rich and dynamic environments where participants can interact with both human and AI teammates in various contexts (Raimondo et al., 2022; Rosero et al., 2021). This allows researchers to study human-machine interactions in controlled yet more realistic settings, leading to a better understanding of constructs such as adaptive teamwork strategies, emergent leadership, conflict resolution, and the impact of AI behavior on human trust and reliance (Bowman et al., 2018).

In HMT research, it is crucial to study how teams adapt to changing circumstances and how individual team members, both human and AI, respond to various challenges. Online games, with their dynamic and interactive nature, provide an ideal platform for such studies. Researchers can design experiments that incorporate varied game scenarios, allowing them to observe and analyze team behaviors and interactions in real-time. This approach not only enhances the ecological validity of the research but also provides valuable insights into the design and implementation of AI agents that can effectively team with humans in complex environments.

For instance, the unpredictability of human behavior in gaming environments can mimic real-world situations where team members must quickly adapt to new challenges and unexpected events. By observing these interactions, researchers can identify patterns and strategies that contribute to successful teamwork. This understanding can then be applied to the development of AI systems designed to operate in dynamic and unpredictable settings, enhancing their ability to collaborate with human teammates.

Improving Ecological Validity of Teaming Scenarios

Ecological validity refers to the extent to which research findings can be generalized to real-world settings (Neisser, 1976). Online games often involve realistic scenarios that closely mimic real-life tasks and interactions with continuous information processing, decision-making, and problem-solving, making them ideal for studying complex team cognitive processes. This level of cognitive realism is crucial for making ecologically valid conclusions from experimental findings, as it ensures that participants’ behaviors and cognitive processes are representative of those in real-world settings even when high-fidelity sensory elements cannot be realistically or ethically simulated (Cooke & Shope, 2004).

Game-based testbeds are particularly effective for studying social and collaborative cognition, which are critical aspects of HMTs. Social cognition involves understanding and interacting with others, while collaborative cognition focuses on how individuals work together to achieve common goals (Bandura, 2001). Games inherently involve social interaction and teamwork, providing a natural context for studying these cognitive processes. Participants in game-based testbeds must communicate, coordinate, and collaborate with both human and AI teammates, simulating the complexities of real-world teamwork. This allows researchers to observe how social and collaborative cognition unfold in dynamic settings, leading to insights into effective teamwork strategies and the design of AI systems that can integrate seamlessly with human teams.

Expanding Recruitment Reach

Recruitment is a critical aspect of any research study, and traditional methods often face significant limitations in terms of reach and diversity. Online gaming environments offer a unique solution to this challenge by providing a global platform that can attract a diverse participant pool. The widespread popularity of online games means that researchers can recruit participants from various demographic and cultural backgrounds, enhancing the generalizability of the research findings (Williams et al., 2008). Moreover, online gaming communities are highly engaged and motivated, making them an ideal target for recruitment. Platforms like Discord, which are dominant among gaming communities, provide excellent channels for reaching out to potential participants. By leveraging these communities, researchers can quickly and efficiently recruit a large number of participants, facilitating the collection of extensive and diverse datasets.

The ability to reach a global audience is particularly valuable for HMT research, which often requires large and varied participant samples to ensure robust and generalizable results. Traditional recruitment methods, such as university email lists and social media posts, may not reach the necessary number or diversity of participants. In contrast, online gaming platforms offer a vast and engaged audience that can be tapped into with relative ease. This broad recruitment reach not only enhances the diversity of the participant pool but also increases the likelihood of obtaining high-quality data that accurately reflects the complexities of human-AI interactions.

Exploring Advanced HMT Concepts

The controlled yet flexible environment of online games enables the investigation of sophisticated HMT concepts. Researchers can design experiments to test various AI behaviors and their effects on human teammates, fostering a deeper understanding of the interplay between people and AI agents. For instance, researchers can explore how different AI behaviors influence human trust, reliance, and overall team performance. Such insights are critical for developing AI agents that can effectively collaborate with humans in various contexts (Johnson et al., 2020).

Additionally, online gaming environments allow researchers to study adaptive teamwork strategies, where both human and AI teammates adjust their behaviors based on changing circumstances. This aspect of HMT is particularly important for developing AI systems that can operate in dynamic and unpredictable environments. By simulating real-life challenges within the game, researchers can observe how teams adapt and respond, providing valuable data for improving AI design and implementation.

The ability to conduct controlled experiments in a realistic and engaging environment is a significant advantage of using online games for HMT research. Researchers can manipulate various factors within the game, such as the behavior of AI agents or the complexity of the tasks, to study their impact on team dynamics and performance. This level of control allows for a detailed examination of specific aspects of human-AI interaction, leading to more precise and actionable insights.

Case Study: Asist Study 4

Throughout the Fall of 2023, our team conducted the fourth and final experiment within the DARPA ASIST program. ASIST was focused on developing and studying artificial social intelligence embedded in virtual testbed environments with social intelligence capabilities to interact with human teammates while completing virtual missions (Fiore et al., 2021). In Study 4, which used the ASIST Dragon testbed (Huang et al., 2024), we developed a Minecraft-based testbed environment that included a bomb location and disposal task that required varying levels of teamwork and coordination throughout the task. The AI agents served as non-embodied advisors and communicated with the human players via browser-based chat and messages within Minecraft during the trial. Our team used this study to examine how AI intervention impacted team processes and outcomes for a team of three humans across conditions with one of two agent advisors or no advisor. Given the complexity of our research questions and the need for enough data to train the agents (i.e., machine learning), we required a large sample size. We relied on online gaming communities and game-based challenges with prize incentives to bring together a large pool of participants.

Our approach capitalized on the potential of online gaming environments, particularly Minecraft, which has a large active user community. We created a customized modification (mod) tailored to our research needs, balancing task design with enjoyable gameplay to maintain an authentic gaming experience while collecting valuable research data. The iterative task design process ensured that experimental goals were met without compromising the engaging nature of the game.

We also leveraged the competitive spirit inherent in gaming by introducing leaderboards and competitive events within the game. This approach not only enhanced the gaming experience but also motivated participants to engage deeply with research tasks, generating richer data. An automated, scalable architecture allowed players to create accounts, join teams, complete surveys, and participate in multiple trials without direct administrator involvement (Yee, 2006). Financial incentives were carefully structured to complement the intrinsic motivations provided by the game, rewarding participation, performance, and consistency to ensure long-term engagement and high-quality data collection. We offered $20 per person for every three games on each game day event to encourage every player to play at least three games; an extra $25 per person for top three teams of each week for competitive players, another $25 for one lucky team within the top 50% of teams for the week excluding the top three of the week; and an extra $50 per person for the top 10 teams across the entire study to encourage participation across multiple weeks.

The use of financial incentives in conjunction with intrinsic motivators provided by the game was a key strategy in maintaining high levels of participant engagement. These incentives were designed to reward not only participation but also performance and consistency, ensuring that participants remained motivated over the course of the study. By carefully balancing intrinsic and extrinsic motivators, we were able to create an environment that encouraged sustained participation.

Findings From ASIST Study 4

Over the course of 3 months of recruitment for ASIST Study 4, we made several key adjustments to improve our data collection. Initially, our efforts yielded modest results, with only 17 trials in the first month. However, three critical changes significantly improved our outcomes. First, we established “game day” events to create a critical mass of players needed to form teams. This strategy addressed the issue of players logging in, finding no available teammates, and subsequently leaving. Second, we refined our recruiting approach by targeting gaming communities on Discord, significantly increasing player engagement. Third, we enhanced financial incentives tied to competition during “game day” events, boosting player participation. These adjustments led to a dramatic increase in games played, from 17 in the first month to 200 per game day event, culminating in nearly 1,200 complete data trials with 257 players (Huang et al., 2024)—one of the largest publicly available HMT datasets in existence.

Discussion and Implications

The effectiveness of online gaming platforms in engaging participants for HMT research underscores the potential of gamification in research methodologies. This approach can lead to richer data collection by ensuring sustained participant involvement and more authentic interactions. The integration of competitive elements and incentives within the gaming environment highlights the importance of motivation in participant engagement. Future research should explore the optimal balance of intrinsic and extrinsic motivators to maximize participant contribution and data quality (Deci & Ryan, 2000).

Community engagement proved valuable in recruiting a diverse and large participant pool. Leveraging existing gaming communities facilitated rapid recruitment and fostered a sense of belonging among participants, enhancing the research experience and outcomes. The use of Discord facilitated an online community of players that would even discuss the game while they were not playing, which was positive for community engagement but had to be monitored to ensure the game was not discussed in-depth with new players revealing information about the task. This work paves the way for a systematic evaluation of online gaming environments for HMT experimentation. Future studies could explore the scalability of gaming platforms for larger studies, the transferability of findings from virtual to real-world settings, and the long-term impacts of gamified research on participant behavior and team dynamics.

The innovative use of online gaming environments in HMT research addresses current methodological challenges and opens new avenues for engaging effective research. By integrating gaming environments into HMT studies, researchers can unlock new insights into the design and implementation of AI agents. This approach not only overcomes existing recruitment and engagement hurdles but also enriches the field’s understanding of the interplay between human and AI teammates.

Game-based testbeds offer a transformative platform for HMT research, providing environments that are not only engaging but also highly realistic at a cognitive level. These environments enhance the ecological validity of studies by simulating real-world scenarios that involve continuous information processing, decision-making, and problem-solving. By effectively managing cognitive load and facilitating flow, game-based testbeds ensure that participants remain engaged and perform optimally. This leads to the collection of authentic data on cognitive processes, which is essential for understanding and improving human-AI interactions. Through their inherent social and collaborative nature, games also provide rich contexts for studying teamwork dynamics, furthering our insights into effective human-machine teaming.