To investigate the relationship between variations in physiological signals and mental workload (MWL) during the execution of a helicopter roll-attitude compensatory tracking task.
Background
Current perceptual models in human–machine piloting have been focused on visual and vestibular cues, overlooking somatosensory and auditory inputs and their interactions. This creates a knowledge gap in understanding shared perception strategies for piloting in environments with impaired sensory channels or enhanced secondary cues.
Methods
Fifteen healthy participants performed an attitude-tracking task under eleven cueing modalities combining visual, degraded visual, haptic, and auditory cues. Physiological signals—cardiac activity, respiration, brain activity, skin temperature, and electrodermal activity—were analyzed in relation to self-reported MWL using statistical tests and GLMM (Generalized Linear Mixed Models).
Results
Participants reported low perceived MWL under good visual conditions, with supplementary auditory and haptic cues helping to reduce MWL with degraded or absent visual input. Physiological signals discriminated between MWL levels and multivariate analysis showed that while combined signals revealed an evident explanation of their variance, individual differences underscored the importance of personalized modeling.
Conclusion
MWL assessment through physiological signals validated during a helicopter tracking task demonstrated that multimodal cueing in complex scenarios can reduce cognitive load, leading to potential safety risk mitigation.
Application
This research has the objective of providing a novel approach for safety enhancement and mitigating risks in rotorcraft operations by integrating visual, auditory, and somatosensory cues with physiological-based MWL assessment.
AmadoriP. V.DemirisY. (2024). User-aware multilevel cognitive workload estimation from multimodal physiological signals. IEEE Transactions on Cognitive and Developmental Systems, 16(4), 1212–1222. https://doi.org/10.1109/TCDS.2023.3342139
2.
AndersC.MoontahaS.RealS.ArnrichB. (2024). Unobtrusive measurement of cognitive load and physiological signals in uncontrolled environments. Scientific Data, 11(1), 1000. https://doi.org/10.1038/s41597-024-03738-7
3.
AngelakiD. E.GuY.DeAngelisG. C. (2009). Multisensory integration: Psychophysics, neurophysiology and computation. Current Opinion in Neurobiology, 19(4), 452–458. https://doi.org/10.1016/j.conb.2009.06.008
4.
BegaultD.WenzelE. M.Godfroy-CooperM.MillerJ. D.AndersonM. R. (2010). Applying spatial audio to human interfaces: 25 years of NASA experience. Proceedings of the 40th international AES conference, Tokyo, Japan, 8 October 2010.
5.
BirdG. D.LauwereynsJ.CrawfordM. T. (2012). The role of eye movements in decision making and the prospect of exposure effects. Vision Research, 60, 16–21. https://doi.org/10.1016/j.visres.2012.02.014
6.
BrillJ. C.LawsonB. D.RupertA. (2014). Tactile situation awareness system (TSAS) as a compensatory aid for sensory loss. Proceedings of the 58th Annual Meeting of the Human Factors and Ergonomics Society, 58(1), 1028–1032. https://doi.org/10.1177/1541931214581215
7.
ButmeeT.LansdownT. C.WalkerG. H. (2019). Mental workload and performance measurements in driving task: A review literature. In SchmorrowD.FidopiastisC. (Eds.), Proceedings of the 20th congress of the International Ergonomics Association (IEA 2018) (Vol. 823, pp. 245–254). Springer. https://doi.org/10.1007/978-3-319-96074-6_31
8.
CharlesR. L.NixonJ. (2019). Measuring mental workload using physiological measures: A systematic review. Applied Ergonomics, 74, 221–232. https://doi.org/10.1016/j.apergo.2018.08.028
9.
ColletC.PetitC.ChampelyS.DittmarA. (2003). Assessing workload through physiological measurements in bus drivers using an automated system during docking. Human Factors, 45(4), 539–548. https://doi.org/10.1518/hfes.45.4.539.27082
10.
ColoniusH.DiederichA. (2001). A maximum-likelihood approach to modeling multisensory enhancement. Advances in neural information processing systems, Vancouver, BC, Canada, 3–8 December 2001. https://doi.org/10.7551/mitpress/1120.001.0001
11.
DansP.FogliaS.NelsonA. (2021). Data processing in functional near-infrared spectroscopy (fNIRS) motor control research. Brain Sciences, 11(5), 606. https://doi.org/10.3390/brainsci11050606
12.
DeldyckeP. J.Van BaelenD.PoolD. M.van PaassenM. M.MulderM. (2018). Design and evaluation of a haptic aid for training of the manual flare manoeuvre. Proceedings of the AIAA SciTech forum (AIAA 2018-0113), Kissimmee, FL, January 2018. https://doi.org/10.2514/6.2018-0113
13.
Dell’AgnolaF.PascucciF.PrioreP. C.JacovittiG.MenniniA.PlacidiG. (2022). Machine-learning-based monitoring of cognitive workload in rescue missions with drones. IEEE Journal of Biomedical and Health Informatics, 26(9), 4751–4762. https://doi.org/10.1109/JBHI.2022.3186625
de StigterS.MulderM.van PaassenM. M. (2007). Design and evaluation of a haptic flight director. Journal of Guidance, Control, and Dynamics, 30(1), 35–46. https://doi.org/10.2514/1.20593
16.
D’IntinoG.OlivariM.BuelthoffH. H.PolliniL. (2020). Haptic assistance for helicopter control based on pilot intent estimation. Journal of Aerospace Information Systems, 17(4), 193–203. https://doi.org/10.2514/1.I010773
17.
D’IntinoG.OlivariM.GeluardiS.FabbroniD.BuelthoffH.PolliniL. (2018). A pilot intent estimator for haptic support systems in helicopter maneuvering tasks. AIAA 2018-0116, proceedings of the AIAA SciTech forum, Kissimmee, FL, 7 January 2018.
18.
EndsleyM. R. (1995). Toward a theory of situation awareness in dynamic systems. Human Factors: The Journal of the Human Factors and Ergonomics Society, 37(1), 32–64. https://doi.org/10.1518/001872095779049543
19.
FabbroniD.GeluardiS.GerboniC. A.OlivariM.PolliniL.BülthoffH. H. (2017). Quasi-transfer of helicopter training from fixed-to motion-base simulator. Proceedings of the 43rd European rotorcraft forum, Milano, Italy, 2017.
20.
FaulF.ErdfelderE.LangA.-G.BuchnerA. (2007). G*Power 3: A flexible statistical power analysis program for the social, behavioral, and biomedical sciences. Behavior Research Methods, 39(2), 175–191. https://doi.org/10.3758/BF03193146
21.
GrantR. C.CarswellC. M.LioC. H.SealesW. B. (2013). Measuring surgeons’ mental workload with a time-based secondary task. Ergonomics in Design: The Quarterly of Human Factors Applications, 21(1), 7–11. https://doi.org/10.1177/1064804612466068
22.
HuangJ.LiuY.PengX. (2022). Recognition of driver’s mental workload based on physiological signals: A comparative study. Biomedical Signal Processing and Control, 71, 103094. https://doi.org/10.1016/j.bspc.2021.103094
23.
JenningsS.CheungB.RupertA.SchultzK.CraigG. (2004). Flight-test of a tactile situational awareness system in a land-based deck landing task. Proceedings of the Human Factors and Ergonomics Society 48th Annual Meeting, 48(1), 563–567. https://doi.org/10.1177/154193120404800131
24.
JeramG. J.PrasadJ. V. R. (2005). Open architecture for helicopter tactile cueing systems. Journal of the American Helicopter Society, 50(3), 238–248. https://doi.org/10.4050/1.3092860
25.
KlydeD. H.LamptonA.MitchellD. G.BerkaC.RhinehartM. (2021). A new approach to aircraft handling qualities prediction. Proceedings of the AIAA Scitech 2021 forum (virtual). https://doi.org/10.2514/6.2021-0178
26.
KlydeD. H.RuckelP.PitoniakS. P.SchulzeP. C.RigsbyJ.XinH.BrewerR.HornJ.FegelyC. E.ConwayF.OttC. R.FellW. C.MulatoR.BlankenC. L. (2018). Piloted simulation evaluation of tracking MTEs for the assessment of high-speed handling qualities. Proceedings of the 74th annual forum of the American helicopter society, Phoenix, AZ, 14 May 2018–17 May 2018.
27.
LamptonA. K.KlydeD. H.MussoD.MitchellD.RhinehartM. (2024). Further development of an approach to aircraft handling qualities prediction. Proceedings of the AIAA Scitech 2024 forum, Orlando, FL, 8–12 January 2024. https://doi.org/10.2514/6.2024-2479
28.
LinY.-A.NobleE.LohC.-H.LohK. J. (2022). Respiration monitoring using a motion tape chest band and portable wireless sensing node. Journal of Commercial Biotechnology, 27(1), 32–38. https://doi.org/10.5912/jcb1026
LongoL. (2017). Subjective usability, mental workload assessments and their impact on objective human performance. In KurosuG. (Ed.), Human-computer interaction – INTERACT 2017 (10514, pp. 115–132). Springer. https://doi.org/10.1007/978-3-319-67684-5_13
31.
LukociusR.VirbalisJ.DaunorasJ.VegysA. (2008). The respiration rate estimation method based on the signal maximums and minimums detection and the signal amplitude evaluation. Elektronika ir Elektrotechnika.
32.
LuzzaniG.BuraioliI.DemarchiD.GuglieriG. (2023). A review of physiological measures for mental workload assessment in aviation: A state-of-the-art review of mental workload physiological assessment methods in human-machine interaction analysis. Aeronautical Journal, 128(1323), 1–22. https://doi.org/10.1017/aer.2023.101
33.
LuzzaniG.BuraioliI.GuglieriG.DemarchiD. (2025). EDA, PPG and skin temperature as predictive signals for mental failure by a statistical analysis on stress and mental workload. IEEE Open Journal of Engineering in Medicine and Biology, 6, 248–255. https://doi.org/10.1109/OJEMB.2024.3515473
34.
MahdaviN.TapakL.DarvishiE.Doosti-IraniA.Shafiee MotlaghM. (2024). Unraveling the interplay between mental workload, occupational fatigue, physiological responses and cognitive performance in office workers. Scientific Reports, 14(1), 17866. https://doi.org/10.1038/s41598-024-68889-4
35.
McGrathB. J.EstradaA.BraithwaiteM. G.RajA. K.RupertA. H. (2004). Tactile situation awareness system flight demonstration. Army Aeromedical Research Laboratory.
36.
MeredithM. A.SteinB. E. (1986). Spatial factors determine the activity of multisensory neurons in cat superior colliculus. Brain Research, 365(2), 350–354. https://doi.org/10.1016/0006-8993(86)91648-3
37.
MillerJ. D.Godfroy-CooperM.SzoboszlayZ. P. (2019). Augmented-reality multimodal cueing for obstacle awareness: Towards a new topology for threat-level presentation. Proceedings of the 75th annual forum of the vertical flight society, Philadelphia, PA, May 2019. https://doi.org/10.4050/F-0075-2019-14562
38.
MohanaveluK.PoonguzhaliS.JananiA.VinuthaS. (2022). Machine learning-based approach for identifying mental workload of pilots. Biomedical Signal Processing and Control, 75, 103623. https://doi.org/10.1016/j.bspc.2022.103623
39.
MorcosM.FishmanS.SaettiU.BergerT.BachelderE. N.Godfroy-CooperM. (2025). Full-body haptic and spatial audio cueing for augmented pilot perception. Journal of Guidance, Control, and Dynamics, 48(6), 1334–1347. https://doi.org/10.2514/1.G008722
40.
MorcosM. T.FishmanS. M.CoccoA.SaettiU.BergerT.Godfroy-CooperM.BachelderE. N. (2023a). Full-body haptic cueing algorithms for augmented pilot perception in degraded/denied visual environments. Proceedings of the 79th annual forum of the vertical flight society, West Palm Beach, FL, May 2023.
41.
MorcosM. T.FishmanS. M.SaettiU.BachelderE. N.Godfroy-CooperM. (2024). Full-body haptic and spatial audio cueing algorithms for augmented pilot perception. Proceedings of the 80th annual forum of the vertical flight society, Montreal, QC, Canada, 7–9 May 2024. https://doi.org/10.4050/F-0080-2024-1179
42.
MorcosM. T.FishmanS. M.SaettiU.BergerT.Godfroy-CooperM.BachelderE. N. (2023b). Spatial audio cueing algorithms for augmented pilot perception in degraded/denied visual environments. Proceedings of the 49th European rotorcraft forum, Bückeburg, Germany, September 2023.
43.
MüllhäuserM.LeißlingD. (2019). Development and In-Flight evaluation of a haptic torque protection. Journal of the American Helicopter Society, 64(1), 012003. https://doi.org/10.4050/JAHS.64.012003
44.
MüllhäuserM.LusardiJ. (2022). US-German joint in-flight and simulator evaluation of collective tactile cueing for torque limit avoidance: Shaker versus soft stop. Journal of the American Helicopter Society, 67(3), 032006. https://doi.org/10.4050/JAHS.67.032006
45.
NakagawaS.SchielzethH. (2013). A general and simple method for obtaining R2 from generalized linear mixed-effects models. Methods in Ecology and Evolution, 4(2), 133–142. https://doi.org/10.1111/j.2041-210X.2012.00261.x
46.
NeigelA. R.DeverD. A.ClaypooleV. L.SzalmaJ. L. (2019). Task engagement and the vigilance decrement revisited: Expanding upon the work of Joel S. Warm using a semantic vigilance paradigm. Human Factors, 61(3), 462–473. https://doi.org/10.1177/0018720819835086
47.
NinoV.MonfortS. M.ClaudioD. (2024). Exploring the influence of individual factors on the perception of mental workload and body postures. Ergonomics, 67(7), 881–896. https://doi.org/10.1080/00140139.2023.2243406
48.
OlivariM.NieuwenhuizenF. M.BülthoffH. H.PolliniL. (2014). Pilot adaptation to different classes of haptic aids in tracking tasks. Journal of Guidance, Control, and Dynamics, 37(6), 1741–1753. https://doi.org/10.2514/1.G000534
49.
OskarssonP. A.ErikssonL.CarlanderO. (2011). Enhanced perception and performance by multimodal threat cueing in simulated combat vehicle. Human Factors, 54(1), 122–137. https://doi.org/10.1177/0018720811424895
50.
PfeiferM.ScholkmannF.LabruyèreR. (2018). Signal processing in functional near-infrared spectroscopy (fNIRS): Methodological differences lead to different statistical results. Frontiers in Human Neuroscience, 11, 641. https://doi.org/10.3389/fnhum.2017.00641
51.
ReynalM.ImbertJ. P.AricoP.ToupillierJ.BorghiniG.HurterC. (2019). Audio focus: Interactive spatial sound coupled with haptics to improve sound source address in poor visibility. International Journal of Human-Computer Studies, 129, 116–128. https://doi.org/10.1016/j.ijhcs.2019.04.001
52.
RoscoeA. H.EllisG. A. (1990). A subjective rating scale for assessing pilot workload in flight: A decade of practical use(RAE Technical Report 90019). Royal Aerospace Establishment.
53.
RyuD.AbernethyB.MannD. L.PooltonJ. M.GormanA. D. (2013). The role of central and peripheral vision in expert decision making. Perception, 42(6), 591–607. https://doi.org/10.1068/p7487
54.
SahasrabudheV.HornJ. F.SahaniN.FaynbergA.SpauldingR. (2006). Simulation investigation of a comprehensive collective-axis tactile cueing system. Journal of the American Helicopter Society, 51(3), 215–224. https://doi.org/10.4050/1.3092883
Taheri GorjiH.WilsonN.VanBreeJ.HoffmannB.PetrosT.TavakolianK. (2023). Using machine learning methods and EEG to discriminate aircraft pilot cognitive workload during flight. Scientific Reports, 13(1), 2507. https://doi.org/10.1038/s41598-023-29647-0
57.
TaoD.TanH.WangH.ZhangX.QuX.ZhangT. (2019). A systematic review of physiological measures of mental workload. International Journal of Environmental Research and Public Health, 16(15), 2716. https://doi.org/10.3390/ijerph16152716
TzafestasS. T.BirbasK.KoumpourosY.ChristopoulosD. (2008). Pilot evaluation study of a virtual paracentesis simulator for skill training and assessment: The beneficial effect of haptic display. Presence: Teleoperators and Virtual Environments, 17(2), 212–229. https://doi.org/10.1162/pres.17.2.212
60.
U.S. Army Aviation and Missile Command. (2000). Aeronautical design standard: Performance specification handling qualities requirements for military rotorcraft(ADS-33E-PRF). Aviation Engineering Directorate.
61.
Van BaelenD.EllerbroekJ.van PassenM. M.MulderM. (2020). Design of a haptic feedback system for flight envelope protection. Journal of Guidance, Control, and Dynamics, 43(4), 700–714. https://doi.org/10.2514/1.G004596
62.
VidulichM. A.TsangP. S. (2012). Mental workload and situation awareness. In SalvendyG. (Ed.), Handbook of human factors and ergonomics (4th ed., pp. 243–273). Wiley. https://doi.org/10.1002/9781118131350.ch8
63.
WeiW.FuX.ZhuY.LuN.MaS. (2023). Classification and prediction of driver’s mental workload based on long time sequences and multiple physiological factors. IEEE Access, 11, 81725–81736. https://doi.org/10.1109/ACCESS.2023.3300721
64.
WenzelE. M.Godfroy-CooperM. (2021). Advanced multimodal solutions for information presentation. NASA TM–20210017507, Ames Research Center.
65.
WickensC. D. (2002). Multiple resources and performance prediction. Theoretical Issues in Ergonomics Science, 3(2), 159–177. https://doi.org/10.1080/14639220210123806
66.
WilsonG. F.RussellC. A. (2003). Real-time assessment of mental workload using psychophysiological measures and artificial neural networks. Human Factors, 45(4), 635–644. https://doi.org/10.1518/hfes.45.4.635.27088
67.
WolfF.KuberR. (2018). Developing a head-mounted tactile prototype to support situational awareness. International Journal of Human-Computer Studies, 109, 54–67. https://doi.org/10.1016/j.ijhcs.2017.08.002
68.
WuX.LiZ. (2013). Secondary task method for workload measurement in alarm monitoring and identification tasks. In RauP. L. P. (Ed.), Cross-cultural design: Methods, practice, and case studies. CCD 2013 (Vol. 8023, pp. 331–340). Springer. https://doi.org/10.1007/978-3-642-39143-9_39
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
