Humans commanding and monitoring robots’ actions are used in various high-stress environments, such as the Predator or MQ-9 Reaper remotely piloted unmanned aerial vehicles. The presence of stress and potential costly mistakes in these environments places considerable demands and workload on the human supervisors, which can reduce task performance. Performance may be augmented by implementing an adaptive workload human–machine teaming system that is capable of adjusting based on a human’s workload state. Such a teaming system requires a human workload assessment algorithm capable of estimating workload along multiple dimensions. A multi-dimensional algorithm that estimates workload in a supervisory environment is presented. The algorithm performs well in emulated real-world environments and generalizes across similar workload conditions and populations. This algorithm is a critical component for developing an adaptive human–robot teaming system that can adapt its interactions and intelligently (re-)allocate tasks in dynamic domains.
AasmanJ.MulderG.MulderL. (1987). Operator effort and the measurement of heart-rate variability. Human Factors, 29, 161–170.
2.
AbbassH. A.TangJ.AminR.EllejmiM.KirbyS. (2014). Augmented cognition using real-time EEG-based adaptive strategies for air traffic control. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 58, 230–234.
3.
ArcherS.GosakanM.ShorterP.LockettJ. (2005). New capabilities of the army’s maintenance manpower modeling tool. Journal of the International Test and Evaluation Association, 26, 19–26.
4.
BacksR. W.WalrathL. C. (1992). Eye movement and pupillary response indices of mental workload during visual search of symbolic displays. Applied Ergonomics, 23, 243–254.
5.
BertholdA.JamesonA. (1999). Interpreting symptoms of cognitive load in speech input. In KayJ. (Ed.), UM99 user modeling (pp. 235–244). New York, NY: Springer.
CainB. (2007). A review of the mental workload literature (Technical Report RTO-TR-HFM-121-PART-II). Toronto: Defence Research and Development Canada.
10.
CastorM. (2003). GARTEUR handbook of mental workload measurement. Madrid, Spain: Group for Aeronautical Research and Technology in Europe.
11.
ChappelleW.ScheimanT.SanfordE. (2018). U.S. air force special operations command remotely piloted aircraft operator fatigue levels and compensatory strategies (Technical Report AFRL-SA-WP-TR-2018-0013). Fairborn, OH: Wright-Patterson Air Force Base.
12.
ChenJ. Y.BarnesM. J. (2014). Human-agent teaming for multi-robot control: A review of human factors issues. IEEE Transactions on Human-Machine Systems, 44, 13–29.
13.
ChristensenJ. C.EsteppJ. R.WilsonG. F.RussellC. A. (2012). The effects of day-to-day variability of physiological data on operator functional state classification. NeuroImage, 59, 57–63. doi:10.1016/j.neuroimage.2011.07.091
14.
CummingsM. L.MastracchioC.ThornburgK. M.MkrtchyanA. (2013). Boredom and distraction in multiple unmanned vehicle supervisory control. Interacting with Computers, 25, 34–47.
15.
DurkeeK. T.HiriyannaA.PappadaS.FeeneyJ.GalsterS. (2016). Multi-model approach to human functional state estimation. In SchmorrowD. D.FidopiastisC. M. (Eds.), Lecture notes in computer science (pp. 188–197). Cham, Switzerland: Springer.
16.
DurkeeK. T.PappadaS. M.OrtizA. E.FeeneyJ. J.GalsterS. M. (2015). System decision framework for augmenting human performance using real-time workload classifiers. In IEEE international multi-disciplinary conference on cognitive methods in situation awareness and decision (pp. 8–13). New York, NY: IEEE. doi:10.1109/COGSIMA.2015.7107968
17.
FunkeG. J.BestC.MenkeL. E.StrangA. J. (2017). Warfighter acceptance of future physio-behavioral monitoring and augmentation: Update. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 61, 1151–1155.
18.
GawronV. J. (2008). Human performance, workload, and situational awareness measures handbook. Boca Raton, FL: CRC Press.
19.
HancockP. A.SzalmaJ. L. (2008). Performance under stress. Surrey, UK: Ashgate.
20.
HankinsT. C.WilsonG. F. (1998). A comparison of heart rate, eye activity, EEG and subjective measures of pilot mental workload during flight. Aviation, Space, and Environmental Medicine, 69, 360–367.
21.
HarriottC. E.ZhangT.AdamsJ. A. (2013). Assessing physical workload for human-robot peer-based teams. International Journal of Human-Computer Studies, 71, 821–837.
22.
HarrisD. (2011). Human performance on the flight deck. Surrey, UK: Ashgate.
23.
HartS. G.StavelandL. E. (1988). Development of NASA-TLX (Task Load Index): Results of empirical and theoretical research. Advances in Psychology, 52, 139–183.
24.
HeardJ.HarriottC. E.AdamsJ. A. (2018). A survey of workload assessment algorithms. IEEE Transactions on Human-Machine Systems, 48, 434–451.
25.
HirshfieldL. M.SoloveyE.GirouardA.KebingerJ.JacobR.SassaroliA.FantiniS. (2009). Brain measurement for usability testing and adaptive interfaces: An example of uncovering syntactic workload with functional near infrared spectroscopy. In Proceedings of the SIGCHI conference on human factors in computing systems (pp. 2185–2194). New York, NY: Association for Computing Machinery. doi:10.1145/1518701.1519035
26.
HooeyB. L.KaberD. B.AdamsJ. A.FongT. W.GoreB. F. (2017). The underpinnings of workload in unmanned vehicle systems. IEEE Transactions on Human-Machine Systems, 48, 452–467.
27.
JamesonA.KieferJ.MüllerC.Großmann-HutterB.WittigF.RummerR. (2010). Assessment of a user’s time pressure and cognitive load on the basis of features of speech. In CrockerM. W.SiekmannJ. (Eds.), Resource-adaptive cognitive processes (pp. 171–204). Cham, Switzerland: Springer.
JornaP. (1993). Heart rate and workload variations in actual and simulated flight. Ergonomics, 36, 1043–1054.
30.
KaberD. B.EndsleyM. R. (2004). The effects of level of automation and adaptive automation on human performance, situation awareness and workload in a dynamic control task. Theoretical Issues in Ergonomics Science, 5, 113–153.
31.
KellerJ.BlessH.BlomannF.KleinbohlD. (2011). Physiological aspects of flow experiences: Skills-demand-compatibility effects on heart rate variability and salivary cortisol. Journal of Experimental Social Psychology, 47, 849–852.
32.
KingmaD.BaJ. (2015). Adam: A method for stochastic optimization. In International conference on learning representations (pp. 8–13). Ithaca, NY: ArXiv. Retrieved from https://arxiv.org/pdf/1412.6980.pdf
33.
LasleyD. J.HamerR. D.DisterR.CohnT. E. (1991). Postural stability and stereo-ambiguity in man-designed visual environments. IEEE Transactions on Biomedical Engineering, 38, 808–813.
34.
LinJ.MatthewsG.Reinerman-JonesL.WohleberR. (2018). Assessing operator psychological states and performance in UAS operations. In International conference on augmented cognition (pp. 131–147). Cham, Switzerland: Springer. doi:10.1007/978-3-319-91470-1_12
35.
MarquartG.CabrallC.de WinterJ. (2015). Review of eye-related measures of drivers’ mental workload. Procedia Manufacturing, 3, 2854–2861.
36.
McCrakenJ.AldrichT. (1984). Implications for operator workload and system automation goals (Technical Report ASI-479-024-84B). Fort Rucker, AL: U.S. Army Research Institution.
37.
MitchellD. (2000). Mental workload and ARL workload modeling tools (Technical Report ARL-TNL-161). Aberden Proving Ground, MD: Army Research Laboratory.
38.
MiyakeS.YamadaS.ShojiT.TakaeY.KugeN.YamamuraT. (2009). Physiological responses to workload change: A test/retest examination. Applied Ergonomics, 40, 987–996.
39.
MizunoT.SakaiT.KawazuraS.AsanoH.AkehiK.MatsunoS.. . . ItakuraN. (2016). Measuring facial skin temperature changes caused by mental work-load with infrared thermography. IEEE Transactions on Electronics, Information and Systems, 136, 1581–1585.
40.
ParasuramanR. (2003). Neuroergonomics: Research and practice. Theoretical Issues in Ergonomics Science, 4(1–2), 5–20.
41.
PaulP.KuijerF. M.VisserB.KemperH. (1999). Job rotation as a factor in reducing physical workload at a refuse collecting department. Ergonomics, 42, 1167–1178.
42.
PopovicD.StikicM.RosenthalT.KlydeD.SchnellT. (2015). Sensitive, diagnostic and multifaceted mental workload classifier (PHYSIOPRINT). In SchmorrowD. D.FidopiastisC. M. (Eds.), Foundations of augmented cognition (pp. 101–111). Cham, Switzerland: Springer.
43.
PutzeF.JarvisJ.SchultzT. (2010). Multimodal recognition of cognitive workload for multitasking in the car. In IEEE conference on pattern recognition (pp. 3748–3751). New York, NY: IEEE. doi:10.1109/ICPR.2010.913
44.
ReimerB.MehlerB.CoughlinJ.GodfreyK.TanC. (2009). An on-road assessment of the impact of cognitive workload on physiological arousal in young adult drivers. In International conference on automotive user interfaces and interactive vehicular applications (pp. 115–118). New York, NY: ACM. doi:10.1145/1620509.1620531
45.
RoscoeA. H. (1992). Assessing pilot workload: Why measure heart rate, HRV and respiration?Biological Psychology, 34, 259–287.
46.
RusnockC.BorghettiB.McQuaidI. (2015). Objective-analytical measures of workload—The third pillar of workload triangulation? In SchmorrowD. D.FidopiastisC. M. (Eds.), Foundations of augmented cognition (pp. 124–135). Cham, Switzerland: Springer.
SheridanT. B. (2011). Adaptive automation, level of automation, allocation authority, supervisory control, and adaptive control: Distinctions and modes of adaptation. IEEE Transactions on Systems, Man, and Cybernetics, 41, 662–667.
49.
StermanM.KaiserD.MannC.SuyenobuB.BeymaD.FrancisJ. (1993). Application of quantitative EEG analysis to workload assessment in an advanced aircraft simulator. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 37, 118–121.
50.
StermanM.MannC. (1995). Concepts and applications of EEG analysis in aviation performance evaluation. Biological Psychology, 40, 115–130.
51.
ThompsonW. T.LopezN.HickeyP.DaLuzC.CaldwellJ. L.TvaryanasA. P. (2006). Effects of shift work and sustained operations: Operator Performance in Remotely Piloted Aircraft (OP-REPAIR) (Technical report). Brooks City-Base, TX: U.S. Air Base.
52.
TingC.MahfoufM.NassefA.LinkensD.PanoutsosG.NickelP.. . . HockeyG. (2010). Real-time adaptive automation system based on identification of operator functional state in simulated process control operations. IEEE Transactions on Systems, Man, and Cybernetics —Part A: Systems and Humans, 40, 251–262.
53.
VicenteK.ThorntonD.MorayN. (1987). Spectral analysis of sinus arrhythmia: A measure of mental effort. Human Factors, 29, 171–182.
54.
WarmJ. S.DemberW. N.HancockP. A. (1996). Vigilance and workload in automated systems. In ParasuramanR.MoulouaM. (Eds.), Automation and human performance: Theory and applications (pp. 183–200). Mahwah, NJ: Lawrence Erlbaum.
55.
WickensC. D.LeeJ. D.LiuY.BeckerS. E. G. (2004). An introduction to human factors engineering (2nd ed.). Upper Saddle River, NJ: Pearson Education.
56.
YinZ.ZhangJ. (2014). Recognition of mental workload levels by combining adaptive exponential feature smoothing and locality preservation projection techniques. In Proceedings of the 33rd Chinese control conference (pp. 4700–4705). New York, NY: IEEE. doi:10.1109/ChiCC.2014.6895732
57.
YoungM. S.StantonN. A. (2002). Malleable attentional resources theory: A new explanation for the effects of mental underload on performance. Human Factors, 44, 365–375.
