I Spy Differences in System Reliability

Abstract

The current experiment examined how the level of exposure to automation of various reliabilities impacts perceptions of reliability and confidence in perceived reliability ratings. Participants viewed images of multicolored Mega blocks presented side-by-side with a recommendation from an automated system indicating whether the images were a match or not. Participants responded whether the automation recommendation was correct or incorrect. The reliability of the automation across blocks was either 50%, 80%, or 90% and participants saw 10, 50, and 100 trials of each reliability. Overall, the level of exposure to the system had no impact on perceptions of automation reliability, or confidence in reliability ratings. Exposure did differentially impact accuracy of automation classification recommendation across the levels of automation reliability. Results also indicated that increases in perceptions of automation reliability increased confidence in the reliability perceptions. These findings have applications for training participants on imperfectly reliable systems.