With the development of eye movement and voice control, multimodal interaction has become a leading trend for future aircraft cockpits. Yet, most studies either demonstrate its effectiveness broadly or focus on single channels, especially the visual channel. A comprehensive ergonomic evaluation method is still lacking.
Objective
This study proposes and validates a model for evaluating multimodal interaction in aircraft cockpits, addressing workload distribution, channel occupancy, and resource demand.
Methods
Our research proposes an evaluation model for multimodal interaction in aircraft cockpits, based on the analysis of pilots’ information cognitive processing. The model sequentially evaluates the workload of multimodal interaction technology in aircraft cockpit scenarios from the perspectives of task level, channel occupancy, and channel resource demand. Validation experiments were conducted using route planning tasks, assessing the workload for tactile interaction modes and multimodal interaction modes.
Results
Results demonstrated the effectiveness of the proposed ergonomics evaluation model, as correlation analysis revealed a significant positive relationship between the model outcomes and subjective workload scores under both tactile and multimodal conditions. While multimodal interaction incurred significantly higher task time and subjective workload compared to tactile interaction, it also enabled more balanced workload distribution across channels, suggesting potential advantages for managing complex multi-channel tasks in future cockpit operations.
Conclusions
The validation experiments confirmed the effectiveness of our ergonomics evaluation model for multimodal interaction in airplane cockpit. It offers a practical tool for early cockpit design by identifying channel conflicts and workload distribution issues before prototype development, thus supporting safer and more efficient multimodal systems.
LaViola JrJJKruijffEMcMahanRP, et al.3D User Interfaces: Theory and Practice. Boston, MA: Addison-Wesley Professional, 2017.
2.
CohenPRDalrympleMMoranDB, et al.Synergistic use of direct manipulation and natural language. In: Proceedings of the SIGCHI Conference on Human factors in Computing Systems, 1989, pp.227–233.
3.
OviattS. Multimodal interactive maps: designing for human performance. Hum–Comput Interact1997; 12: 93–129.
4.
Van ErpJBVan VeenH, et al.A multi-purpose tactile vest for astronauts in the international space station. In: Proceedings of eurohaptics. Dublin, Ireland: ACM, Press, 2003, pp.405–408.
5.
WilkinsPActonW. Noise and accidents—a review. Ann Occup Hyg1982; 25: 249–260.
6.
JansenCWennemersAVosW, et al.Flytact: a tactile display improves a helicopter pilot’s landing performance in degraded visual environments. In: Haptics: Perception, Devices and Scenarios: 6th International Conference, EuroHaptics 2008 Madrid, Spain, June 10–13, 2008 Proceedings 6, 2008, pp.867–875: Springer.
7.
ShreeDVJMurthyLRDSalujaK, et al.Operating different displays in military fast jets using eye gaze tracker. J Aviat Technol Eng2018; 8: 31.
8.
RakkolainenIFarooqAKangasJ, et al.Technologies for multimodal interaction in extended reality—a scoping review. Multimodal Technol Interact2021; 5: 81.
9.
LuoTZhangMPanZ, et al.Dream-Experiment: a MR user interface with natural multi-channel interaction for virtual experiments. IEEE Trans Vis Comput Graph2020; 26: 3524–3534.
10.
LuSAWickensCDSarterNB, et al.Informing the design of multimodal displays: a meta-analysis of empirical studies comparing auditory and tactile interruptions. Proc Hum Factors Ergon Soc Annu Meet2011; 55: 1170–1174.
11.
BischoffRGraefeV. Dependable multimodal communication and interaction with robotic assistants. In: Proceedings. 11th IEEE International Workshop on Robot and Human Interactive Communication, 2002, pp.300–305: IEEE.
12.
JohnsonDOAgahA. Human robot interaction through semantic integration of multiple modalities, dialog management, and contexts. Int J Soc Robot2009; 1: 283–305.
13.
WeiZZhuangDWanyanX, et al.A model for discrimination and prediction of mental workload of aircraft cockpit display interface. Chin J Aeronaut2014; 27: 1070–1077.
14.
ZengZSunYLiuX, et al.Visual cognition-based optimised design of primary flight displays in cockpits. Aeronaut J2025; 129: 609–626.
15.
CausseMBehrendJMumawRJ. Aerospace Psychology and Human Factors: Applied Methods and Techniques. Göttingen: Hogrefe, 2024.
16.
TippeyKGRoadyTRodriguez-ParasC, et al.General aviation weather alerting: the effectiveness of different visual and tactile display characteristics in supporting weather-related decision making. Int J Aerosp Psychol2017; 27: 121–136.
17.
StricklandDPioroBNtuenC. The impact of cockpit instruments on pilot exhaustion. Comput Ind Eng1996; 31: 483–486.
18.
SchreuderMRiccioARisettiM, et al.User-centered design in brain–computer interfaces—A case study. Artif Intell Med2013; 59: 71–80.
JinMLuGChenF, et al.Modeling takeover behavior in level 3 automated driving via a structural equation model: considering the mediating role of trust. Accid Anal Prev2021; 157: 106156.
21.
LiangSYuL. Voice search behavior under human–vehicle interaction context: an exploratory study. Libr Hi Tech2024; 42: 1496–1516.
22.
HanYAbowdGDStaskoJ. Intivisor: a visual analytics system for interaction log analysis. IEEE Trans Vis Comput Graph2024; 31: 1772–1784.
23.
LouXFuLYanL, et al.Distance effects on visual search and visually guided freehand interaction on large displays. Int J Ind Ergon2022; 90: 103318.
24.
KumarNKumarJ. Measurement of cognitive load in HCI systems using EEG power spectrum: an experimental study. Procedia Comput Sci2016; 84: 70–78.
25.
LiYLiKWangS, et al.Pilot behavior recognition based on multi-modality fusion technology using physiological characteristics. Biosensors2022; 12: 04.
26.
GuerinoGCValentimNMC. Usability and user experience evaluation of natural user interfaces: a systematic mapping study. IET softw2020; 14: 451–467.
27.
RoiderFRümelinSPflegingB, et al.Investigating the effects of modality switches on driver distraction and interaction efficiency in the car. J Multimodal User Interfaces2019; 13: 89–97.
28.
LiWWuYZengG, et al.Multi-modal user experience evaluation on in-vehicle HMI systems using eye-tracking, facial expression, and finger-tracking for the smart cockpit. Int J Veh Perform2022; 8: 429–449.
29.
TriantafyllidisEMcgreavyCGuJ, et al.Study of multimodal interfaces and the improvements on teleoperation. IEEE Access2020; 8: 78213–78227.
30.
LiW-CZakarijaMYuC-S, et al.Interface design on cabin pressurization system affecting pilot’s situation awareness: the comparison between digital displays and pointed displays. Hum Factors Ergon Manuf Serv Ind2020; 30: 103–113.
31.
StefanAChristophAHaraldR, et al.Aircraft cockpit interaction in virtual reality with visual, auditive, and vibrotactile feedback. Proc ACM Hum-Comput InteractEpub ahead of print 31 October 2023; 7: 420–443.
32.
WangYLiW-CKorekWT, et al.Future flight deck design: developing an innovative touchscreen inceptor combined with the primary flight display. Int J Ind Ergon2024; 101: 103588.
33.
XinYKamKQinbiaoL, et al.Exploring the human-centric interaction paradigm: augmented reality-assisted head-up display design for collaborative human-machine interface in cockpit. Adv Eng Inf2024; 62: 102656.
34.
DongYLiuW. Research on UX evaluation method of design concept under multi-modal experience scenario in the earlier design stages. Int J Int Des Manuf (IJIDeM)2018; 12: 505–515.
BaddeleyA. Exploring Working Memory: Selected works of Alan Baddeley. Routledge, 2017.
38.
AlaisDNewellFMamassianP. Multisensory processing in review: from physiology to behaviour. Seeing Perceiving2010; 23: 3–38.
39.
BierbaumCRSzaboSMAldrichTB. Task analysis of the UH-60 mission and decision rules for developing a UH-60 workload prediction model: Summary report. US Army Research Institute for the Behavioral and Social Sciences, 1989.
40.
McCrackenJHAldrichTB. Analyses of Selected LHX Mission Functions: Implications for Operator Workload and System Automation Goals. Fort Belvoir, VA: Defense Technical Information Center. Epub ahead of print 1 June 1984. DOI: 10.21236/ADA232330.
41.
HuangYZhangNLiZ. Generalizability of mental workload prediction using VACP scales in different fields. In: HarrisDLiW-C (eds) Engineering psychology and cognitive ergonomics. Cham: Springer Nature Switzerland, 2023, pp.79–94.
42.
FernándezMÁPeláezVLópezG, et al.Multimodal interfaces for the smart home: findings in the process from architectural design to user evaluation. In: BravoJLópez-de-IpiñaDMoyaF (eds) Ubiquitous computing and ambient intelligence. Berlin, Heidelberg: Springer, 2012, pp.173–180.
43.
OviattSCoulstonRLunsfordR. When do we interact multimodally? Cognitive load and multimodal communication patterns. In: Proceedings of the 6th International Conference on Multimodal Interfaces, New York, NY, USA: Association for Computing Machinery, pp.129–136.
44.
DrorIE, et al.The paradox of human expertise: why experts get it wrong. Paradoxical Brain2011: 177–188.
45.
EricssonKASmithJ. Toward a general theory of expertise: Prospects and limits. New York, NY, USA: Cambridge University Press, 1991.
