Abstract
What performance level must automation reach to be a net benefit to the user? This paper presents a meta-analysis of 34 data points taken from 12 studies in the human factors literature, each representing the effect of an imperfect automation aid on system performance, relative to baseline. Bayesian regression analysis indicated a consistent relationship between automation reliability (i.e., overall percent correct) and performance, with values greater than 67% associated with performance gains. The credible interval for this crossover point ranged from 55 to 75%. There was also a consistent effect of d’, with a crossover point of 1.47 and a credible interval from -0.04 to 2.22. However, we urge caution in using these values as a benchmark criterion, due to the sizeable uncertainty in the crossover estimates and the variability in how researchers compute false alarm and reliability rates. The question “How good is good enough?” likely does not have a single domain-general answer, with the automation performance threshold varying across task domains and other variables.
Get full access to this article
View all access options for this article.