Beyond Expert Ratings: Subjective Workload and SDT Metrics for Team Performance in VR Simulations

Abstract

Effective team communication and cooperation skills are crucial in safety-critical environments, where errors can have devastating consequences including loss of life. This study evaluated the efficacy of using virtual reality (VR) simulations to measure team performance beyond traditional expert ratings and questionnaires. Utilizing a custom-built VR simulator, this work looked at how varying levels of task difficulty and task interdependence requirements impact reported team workload, error rates, and communication patterns among 20 teams of three (n = 60). Objective metrics investigated included quantity of verbal utterances, error rates, sensitivity (d′), and criterion bias (c) derived from signal detection theory (SDT), while perceived team workload was assessed using the Team Workload Questionnaire (TWLQ). Findings showed that increased task demand characteristics significantly elevated perceived team workload and error rates while degrading overall task performance and decision-making, leading to a strategy shift. This was evidenced by marked drops in sensitivity and shifts toward a conservative response bias along with an increase in attempted verbal communication. Integrating workload assessments alongside objective error and SDT-based performance measures provides a robust framework for evaluating teams within VR simulations, providing an alternative to traditional team performance evaluation methods.