Joint Activity Testing: Towards a Multi-Dimensional,High-Resolution Evaluation Method for Human-Machine Teaming

Abstract

Quantitative evaluations of human-machine teams (HMTs) are desperately needed to ensure technological implementations are helpful rather than harmful to overall system performance; however, as machines increasingly behave like active cognitive teammates, traditional evaluation strategies risk overestimating HMT capabilities. Areliable HMT evaluation method should include multiple high-resolution, continuous measures for both system performance and system challenges that can be implemented unobtrusively in real-time operations. In our prior work, we proposed joint activity testing (JAT) as acandidate evaluation framework to satisfy these requirements. Preliminary efforts with asingle dimension of performance and challenge have indicated that the method can identify the additive benefits of joint activity with aspecific technology. In this paper, we explore the operationalization of multi-dimensional JAT by synthesizing our work in two intelligence and two healthcare domains. The patterns observed between domains will guide future JAT, reveal paths towards real-time implementation, and spark future research evaluating resilience.