VirTLab-Eval: Human-Agent Team and Digital Twin Performance Evaluation Demonstration

Abstract

The study of human-artificial intelligence (AI) teaming (HAT) and Human Digital Twin (HDT) modeling currently faces significant challenges in accurately simulating and assessing the effectiveness of interactions between humans and AI systems. Current methods typically rely on limited real-world data or simplified simulated representations that do not capture the complexity and variability of human digital twin and AI agent behavior. VirTLab-Eval is a novel agentic framework of human digital twins (HDTs) and AI agents to model and evaluate HAT behaviors and interventions across operational scenarios. A comprehensive set of measures, including novel socio-cognitive-emotional and behavioral metrics automatically extracted from team communications and interactions, for evaluating HAT performance is integrated into visual analytics enabling the assessment of interventions that target the attributes of HDTs and agents. VirTLab-Eval measurements capture nuanced aspects of trust development, emotional alignment, and team cohesion that conventional performance metrics might miss, providing richer insights into the HAT dynamics. Example results from search and rescue missions indicate that AI teammate reliability has a significant effect on communication dynamics and assistance behaviors, while Human Digital Twin (HDT) personality traits shape trust development and team coordination. These insights directly inform the design of Human-AI Teaming (HAT) training programs aimed at optimizing the use of AI systems, mitigating both over-reliance and under-utilization, and ultimately enhancing mission effectiveness while reducing risk to personnel.