Building an LLM-Based Teammate in Minecraft: A Testbed for Human-AI Collaboration

Abstract

Commercial-Off-The-Shelf (COTS) games and Large Language Models (LLMs) are enabling new empirical paradigms in the study of human-machine teaming (HMT). COTS games that allow for modifications are lowering barriers to the design and conduct of controlled experimental testbeds, while advances in LLMs have dramatically broadened the scope of possible interaction modes between humans and machines. In this paper, we present the iterative design and development of a Minecraft-based tower defense testbed to investigate the impacts of agent and team composition on HMT performance and team processes. Our study builds on insights from the DARPA Artificial Social Intelligence for Successful Teams (ASIST) in designing two versions of LLM-enabled team-mates that work alongside humans in a task-performer role. We developed our testbed with interactive agents for real-time, action-oriented human-agent teaming. Our focus is on the iterative design and implementation of the testbed, including game design trade-offs between difficulty and performance measurement, as well as our approach to conducting remote experimental data collection.