Ergonomics evaluation modeling of multimodal interaction technology for airplane cockpit and experimental validation

Abstract

Background

With the development of eye movement and voice control, multimodal interaction has become a leading trend for future aircraft cockpits. Yet, most studies either demonstrate its effectiveness broadly or focus on single channels, especially the visual channel. A comprehensive ergonomic evaluation method is still lacking.

Objective

This study proposes and validates a model for evaluating multimodal interaction in aircraft cockpits, addressing workload distribution, channel occupancy, and resource demand.

Methods

Our research proposes an evaluation model for multimodal interaction in aircraft cockpits, based on the analysis of pilots’ information cognitive processing. The model sequentially evaluates the workload of multimodal interaction technology in aircraft cockpit scenarios from the perspectives of task level, channel occupancy, and channel resource demand. Validation experiments were conducted using route planning tasks, assessing the workload for tactile interaction modes and multimodal interaction modes.

Results

Results demonstrated the effectiveness of the proposed ergonomics evaluation model, as correlation analysis revealed a significant positive relationship between the model outcomes and subjective workload scores under both tactile and multimodal conditions. While multimodal interaction incurred significantly higher task time and subjective workload compared to tactile interaction, it also enabled more balanced workload distribution across channels, suggesting potential advantages for managing complex multi-channel tasks in future cockpit operations.

Conclusions

The validation experiments confirmed the effectiveness of our ergonomics evaluation model for multimodal interaction in airplane cockpit. It offers a practical tool for early cockpit design by identifying channel conflicts and workload distribution issues before prototype development, thus supporting safer and more efficient multimodal systems.