Automation is an important and widely utilized component in work environments across many domains; it is useful for completing tasks too dangerous or cumbersome for personnel to complete by themselves. Despite its benefits, potential issues can occur that may impact safety and efficiency in the overall human-automation system. To realize the benefits of automation, designers must be able to measure and assess the levels of safety and efficiency. This paper will discuss a theoretical framework to guide the development and selection of metrics for assessing human-automation interaction.
ArthurW.Jr.BennettW.Jr. (1998). Factors that influence skill decay and retention. Human Performance, 11(1), 57-101.
3.
BurkeC.S.LumH.C.ScielzoS.A.Smith-JentschK.SalasE. (2009). Examining measures of team cognition in virtual teams: A heuristic and guidelines. In SchmorrowD.CohnJ.NicholsonD. (Eds.), The PSI Handbook of Virtual Environments for Training and Education: Developments fo the Military and Beyond. (266-283). Westport, CT: Praeger Security International.
4.
CullenR.H.RogersW.A.FiskA.D. (2013). Human performance in a multiple-task environment: Effects of automation reliability on visual attention allocation. Applied Ergonomics, 44(6), 962-968.
5.
DonmezB.PinaP.E.CummingsM.L. (2008). Evaluation criteria for human-automation performance metrics. In MadhavanR.TunstelE.MessinaE. (Eds.), Performance evaluation and benchmarking of intelligent systems (21-40). Boston, MA: Springer.
6.
DzindoletM.T.PetersonS.A.PomrankyR.A.PierceL.G.BeckH.P. (2003). The role of trust in automation reliance. International Journal of Human-Computer Studies, 58(6), 697-718.
7.
EndsleyM.R. (1995a). Toward a theory of situation awareness in dynamic systems. Human Factors, 37(1), 32-64.
8.
EndsleyM.R. (1995b). Measurement of situation awareness in dynamic systems. Human Factors, 37, 65-84.
9.
EndsleyM.R.KaberD.B. (1999). Level of automation effects on performance, situation awareness and workload in a dynamic control task. Ergonomics, 42(3), 462-492.
10.
EricssonK.A.LehmannA.C. (1996). Expert and exceptional performance: Evidence of maximal adaptation to task constraints. Annual Review of Psychology, 47, 273-305.
11.
FioreS.M.RosenM.A.Smith-JentschK.A.SalasE. (2010). Toward an understanding of macrocognition in teams: Predicting processes in complex collaborative contexts. Human Factors, 52(2), 203-244.
GoldbergL.R. (2001). International personality item pool: A scientifically collaborator for the development of advanced measures of personality traits and other individual differences. Retrieved from http://ipip.ori.org/
14.
GoodrichM.A.OlsenD.R.Jr. (2003). Seven principles of efficient human robot interaction. IEEE International Conference on Systems, Man and Cybernetics, 4, 3942-3948.
15.
HuangF.HwangS. (2009). Experimental studies of computerized procedures and team size in nuclear power plant operations. Nuclear Engineering and Design, 239, 373-380.
16.
HancockP.A.BillingsD.R.SchaeferK.E.ChenJ.Y.C.de VisserE.J.ParasuramanR. (2011). A meta-analysis of factors affecting trust in human-robot interaction. Human Factors, 53, 517-527.
17.
HartS.G.StavelandL.E. (1988). Development of NASA-TLX (Task Load Index): Results of empirical and theoretical research. Advances in Psychology, 52, 139-183.
18.
KaberD.B.EndsleyM.R. (1997). Out-of-the-loop performance problems and the use of intermediate levels of automation for improved control system functioning and safety. Process Safety Progress, 16(3), 126-131.
19.
LeeJ. D.SeeK. A. (2004). Trust in automation: Designing for appropriate reliance. Human Factors: The Journal of the Human Factors and Ergonomics Society, 46(1), 50-80.
20.
MatthewsG.DearyI.J.WhitemanM.C. (2003). Personality Traits. Cambridge, United Kingdom: Cambridge University Press.
21.
MayerR.C.DavisJ.H.SchoormanF.D. (1995). An integrative model of organizational trust. Academy of Management Review, 20, 709-734.
22.
MetzgerU.ParasuramanR. (2001). Automation-related “complacency”: Theory, empirical data, and design implications. In Proceedings of the Human Factors and Ergonomics Society 45th Annual Meeting.
23.
MustafizS.SunX.KienzleJ.VangheluweH. (2008). Model-driven assessment of system dependability. Software & Systems Modeling, 7 (4), 487-502.
24.
OlsenD.R.GoodrichM.A. (2003). Metrics for evaluating human-robot interactions. In Proceedings of the Workshop on Performance Metrics for Intelligent Systems. Gaithersburg, MA.
25.
OakleyB.MoulouaM.HancockP.A. (2003). Effects of automation reliability on human monitoring performance. Proceedings of the Human Factors and Ergonomic Society, 47, 188-190.
26.
ParasuramanR. (2000). Designing automation for human use: empirical studies and quantitative models. Ergonomics, 43(7), 931-951.
27.
ParasuramanR.ManzeyD. H. (2010). Complacency and bias in human use of automation: An attentional integration. Human Factors: The Journal of the Human Factors and Ergonomics Society, 52(3), 381-410.
28.
ParasuramanR.MoulouaM.HilburnB. (1999). Adaptive aiding and adaptive task allocation enhance human-machine interaction. Automation technology and human performance: Current research and trends, 119-123.
29.
ParasuramanR.MoulouaM.MolloyR. (1996). Effects of adaptive task allocation on monitoring of automated systems. Human Factors: The Journal of the Human Factors and Ergonomics Society, 38(4), 665-679.
30.
ParasuramanR.RileyV. (1997). Humans and automation: use, misuse, disuse, abuse. Human Factors: The Journal of the Human Factors and Ergonomics Society, 39(2), 230-253.
31.
ParasuramanR.SheridanT. B.WickensC. D. (2000). A model for types and levels of human interaction with automation. Systems, Man and Cybernetics, Part A: Systems and Humans, IEEE Transactions on, 30 (3), 286-297.
32.
ParasuramanR.SheridanT.WickensC. (2008). Situation awareness, mental workload, and trust in automation: Viable, empirically supported cognitive engineering constructs. Journal of Cognitive Engineering and Decision Making, 2 (2), 140-160.
33.
RiceS.Keller, (2009). Automation reliance under time pressure. Cognitive Technology, 14(1), 36-44.
34.
RoseC.L.MurphyL.B.ByardL.NikzadK. (2002). The role of big five personality factors in vigilance performance and workload. European Journal of Personality, 16, 185-200.
35.
SheridanT.ParasuramanR. (2005). Human-automation interaction. Reviews of human factors and ergonomics, 1(1), 89-129.
36.
SteinfeldA.FongT.KeberD.LewisM.ScholtzJ.SchultzA.GoodrichM. (2006). Common metrics for human-robot interaction.
37.
SzalmaJ.L.TaylorG.S. (2011). Individual differences in response to automation: The five factor model of personality. Journal of Experimental Psychology: Applied, 17(2), 71-96.
38.
VeltmanJ.A.GaillardA.W.K. (1996). Physiological indices of workload in a simulated flight task. Biological Psychology, 42(5), 323-342.
39.
VidulichM. A.WickensC. D. (1986). Causes of dissociation between subjective workload measures and performance: Caveats for the use of subjective assessments. Applied Ergonomics, 17 (4), 291–296.
40.
WickensC.D. (2002). Multiple resources and performance prediction. Theoretical Issues in Ergonomics Science, 3(2), 159-177.
41.
WiegmannD.A.RichA.ZhangH. (2001). Automated diagnostic aids: The effects of aid reliability on users’ trust and reliance. Theoretical Issues of Ergonomic Science, 2(4), 352-367.
