Analyzing Sub-Optimal Human-Automation Performance across Multiple Sessions

Abstract

Objective: The purpose of the current study was to analyze the interaction between operator strategy and consistency over time. Background: Sub-optimal human-automation performance is a phenomenon whereby combining human operators with diagnostic aids results in performance that is less than desirable. Rice, Trafimow and Hunt (in press) used Potential Performance Theory (PPT) to show that strategies account for little of the variance in sub-optimal performance, and that most of the decrements are due to operators inconsistently using these strategies. The current study looks at these issues across multiple sessions. Method: 20 participants spent four sessions searching for enemy helicopters in aerial images of Baghdad; the task was augmented by a 70% reliable diagnostic aid that provided recommendations during each trial. Results: Consistency accounted for improvement in observed scores early on, despite no gains to strategy, while improvements in strategy accounted for increased observed performance in the later sessions, despite no gains to consistency. Individual data were also analyzed separately, showing various reasons for improvement across time. Conclusion: Both consistency and strategy play important roles in observed performance during human-automation interaction. Application: This study points to a topic that designers and users of automated systems should carefully consider.