Are Well-Calibrated Users Effective Users? Associations Between Calibration of Trust and Performance on an Automation-Aided Task

Abstract

Objective:

We present alternative operationalizations of trust calibration and examine their associations with predictors and outcomes.

Background:

It is thought that trust calibration (correspondence between aid reliability and user trust in the aid) is a key to effective human–automation performance. We propose that calibration can be operationalized in three ways. Perceptual accuracy is the extent to which the user perceives the aid’s reliability accurately at one point in time. Perceptual sensitivity and trust sensitivity reflect user adjustment of perceived reliability and trust as the aid’s actual reliability changes over time.

Method:

One hundred fifty-five students completed an X-ray screening task with an automated screener. Awareness of the aid’s accuracy trajectory and error type was examined as predictors, and task performance and aid failure detection were examined as outcomes.

Results:

Awareness of accuracy trajectory was significantly associated with all three operationalizations of calibration, but awareness of error type was not when considered in conjunction with accuracy trajectory. Contrary to expectations, only perceptual accuracy was significantly associated with task performance and failure detection, and combined, the three operationalizations accounted for only 9% and 4% of the variance in these outcomes, respectively.

Conclusion:

Our results suggest that the potential importance of trust calibration warrants further examination. Moderators may exist.

Application:

Users who were better able to perform the task unaided were better able to identify and correct aid failure, suggesting that user task training and expertise may benefit human–automation performance.