Generalizing Predictive Models of Situation Awareness to Unseen Participants

Abstract

Monitoring pilot performance is critical in commercial aviation, where mistakes can lead to high consequences. Situation awareness (SA) is particularly important because it is strongly correlated with piloting performance and could be used to intervene and prevent disaster. This work demonstrates the challenge of predicting SA for pilots that are excluded from model-training and evaluates how much operator information is required for state-of-the-art predictive performance. Written informed consent was collected from 31 participants in a protocol approved by the Institutional Review Board for the University of Colorado Boulder. Participants performed a complex task and responded to objective SA assessments. Their scores were predicted from cognitive, physiological, and behavioral measures captured during the task. ANOVA results (F(30,299) = 17, p << .05) suggest that inter-individual differences account for much of the variance in SA scores. When models are evaluated on unseen participants, errors increase. Standardized mean absolute errors grow from 0.74, 0.76, and 0.69 for perception, comprehension, and projection scores respectively to 1.01, 1.06, and 0.92 when participants were left out from training entirely, but improve when incorporating operator background information. SA monitoring remains dependent on access to individual training data, but operator background information can mitigate this limitation.