Comprehensive Evaluation of Explanation Types in a Spaceflight-Relevant Human–Autonomy Teaming Task

Abstract

Objective

This study evaluates how explanation type in an explainable AI (XAI) human–autonomy teaming (HAT) task affects performance, workload, trust, situation awareness (SA), and preference in a dynamic, spaceflight-relevant simulator. Second, we introduce a holistic evaluation method for comparing XAI systems across multiple outcomes.

Background

XAI aims to improve understanding, calibrate trust, and enhance performance of an HAT, but the impact of explanation type in realistic, high-taskload HAT settings remains underexplored.

Method

Participants ( $N=31$ ) completed 18 trials in a dual-task simulator requiring manual rover driving while supervising an autonomous exploration agent. Participants received various combinations of global, contrastive, and deductive explanations for AI-generated routes, with incentives tied to performance.

Results

Explanation type significantly affected manual performance ( $p=0.0003$ ), autonomy performance ( $p<0.0001$ ), team performance ( $p<0.0001$ ), workload ( $p<0.0001$ ), trust ( $p<0.0001$ ), and preference ( $p=0.001$ ), but not SA ( $p=0.41$ ). Participants preferred global and contrastive explanations, performing better with their preferred explanation ( $p=0.049$ ).

Conclusion

Explanation type influences performance and perception in demanding HAT contexts. A standardized, multi-metric evaluation framework is essential for understanding tradeoffs in XAI design.

Application

In HAT tasks like space exploration where users must quickly make decisions with an AI teammate, designers must consider the explanation method for XAI explanations. Our human-centered evaluation found a contrastive + global explanation combination was the best in our HAT task across a range of performance and preference metrics.