Multimodal Feedback to Improve Performance of Order Picker Truck Drivers Using a Virtual Reality Simulator

Abstract

VR technology presents exciting prospects for immersive forklift training environments. VR facilitates hands-on learning without the risks of real world errors. Real-time multimodal feedback (such as visual and haptic) is an effective way to enhance learning outcomes and skill acquisition. However, the efficacy and usability of real-time multimodal feedback provided for VR-based forklift driving needs to be investigated, as its impact on training outcomes remains unexplored. We aim to understand how various feedback modalities influence task performance, perceived workload, and user preferences. Fifteen (3 female, 12 male) individuals participated and completed VR-based forklift driving tasks with four feedback conditions (No feedback, visual, haptic, combined visual and haptic). A significant main effect of Feedback was found for completion time. Mental demand and frustration were affected by the Sex and Feedback interaction. These differences in the perception of feedback modalities underscore the importance of considering diverse user demographics when designing feedback systems.

Order picker truck drivers play a critical role in warehouse operations yet face challenges in safety, productivity, and performance (Bureau of Labor Statistics, U.S. Department of Labor, 2023). Effective training is essential to mitigate risks and enhance efficiency. While traditional training methods can be costly and resource-intensive (Islam et al., 2023), Virtual reality (VR) technology offers a promising solution by providing immersive and safe training environments (Naranjo et al., 2020). VR simulations can replicate real-world warehouse scenarios, allowing drivers to practice skills and develop proficiency without the risks associated with real-world errors (Zawadzki et al., 2019). Current VR simulators have not yet fully replicated the high-level maneuvering skills and strategies that experienced truck drivers possess, presenting an opportunity for further development in this area (Sarupuri et al., 2016).

Real-time feedback from experienced truck drivers can establish best practices for novices, enhancing learning efficiency, safety awareness, and skill development. Designing such feedback from experienced drivers in a multimodal feedback format, including visual and haptic cues, could improve learning and skill acquisition as previous studies found them effective in assembly and vehicle driving (Islam & Lim, 2022). In VR-based driving simulators, multimodal feedback enhanced realism, immersion, and training effectiveness (Piechowski et al., 2020). However, balancing feedback to avoid overwhelming users with distractions is crucial (Birrell et al., 2013). Research is needed to optimize multimodal feedback for VR-based forklift training, ensuring maximum efficacy without compromising user experience or performance. Thus, we evaluated the effectiveness of multimodal feedback in a VR simulator for forklift driving, aiming to improve the performance of order picker truck drivers. Our goal was to investigate the impact of different feedback methods on task execution, perceived workload, and user preferences.

We recruited a convenience sample of fifteen college students (3 female, 12 male) for our study. Female participants had a mean age of 21.7 years (SD = 5.8), while male participants averaged 22.9 years (SD = 3.9). We used a VR-based order picker forklift simulator (The Raymond Corporation, NY, USA), with physical controls and a VR environment featuring multiple truck lessons ( Islam et al., 2024). Prior to the main experiment, participants underwent a 10-min training session. They completed two driving lessons focused on fork-pallet engagement (L1) and object picking (L2). We integrated various visual and haptic feedback cues based on input from experienced forklift drivers into the VR simulation. During each driving lesson, participants experienced four Feedback Conditions (No Feedback (NF), Visual only (V), Haptic Only (H), and combined Visual and Haptic (VH) in a partially counterbalanced manner. We measured completion time, NASA-Task Load Index (TLX) scores for Mental Demand and Frustration (Hart & Staveland, 1988), and System Usability Scale (SUS) (Bangor et al., 2008) responses, followed by a semi-structured interview. Repeated-measures ANOVAs assessed the effects of Biological Sex and Feedback Conditions on the variables mentioned.

A significant main effect of Feedback was observed for completion time in L1 only [F (3, 60) = 4.1, p = .013]. Participants achieved the fastest completion with haptic feedback, with the only significant difference found between H and VH conditions. In L1, the interaction between Sex and Feedback influenced both mental demand and frustration. Female participants reported higher mental demand and frustration with Visual (V) feedback compared to Haptic (H) feedback. There were no significant differences in the Overall SUS score for either L1 or L2. In semi-structured interviews, participants expressed a preference for V and VH feedback, considering them effective, efficient, and comfortable. Overall, V feedback was favored for its balanced information delivery without overwhelming users.

Our study investigated how V, H, and VH feedback impact participant performance and experience in VR forklift driving. Sex differences in feedback perception highlight the need for inclusive feedback system design considering the driver’s sex. In L1, we found a significant interaction between feedback type and sex for mental demand, suggesting that H feedback may reduce cognitive load, especially for female participants, during initial forklift tasks. This aligns with prior research emphasizing haptic feedback’s cognitive benefits in virtual environments for females (Macias-Velasquez et al., 2024). Additionally, females reported higher frustration in L1 with V feedback compared to H feedback. This indicates that H feedback might mitigate frustration, particularly among female users. Conversely, H feedback was associated with lower frustration levels, potentially enhancing engagement and satisfaction during initial forklift tasks. However, in L2, no significant differences were found in mental demand or frustration across feedback conditions, suggesting that adding feedback did not change the perceived mental demand or frustration. Considering L2 was a more complex task requiring higher precision driving than L1, task complexity should be considered in designing proper feedback as it might not be effective in combination with a complex task (Choi et al., 2020; Froland et al., 2023; Martin et al., 2012).

Objective performance measures generally showed no significant differences between feedback types, except for completion time in L1, where H feedback led to shorter completion times compared to VH condition. Participants noted that H feedback, whether unimodal or multimodal, could be distracting. Overall, participants felt that additional feedback beyond the default simulator settings enhanced task efficiency and reduced perceived difficulty. The consistent SUS scores between L1 and L2 indicate that task variations did not significantly affect overall user evaluations of the training system’s effectiveness and usability.

In summary, our study underscored the importance of user control, technological integration, feedback optimization, and visualization techniques in enhancing VR-based training. These insights can inform the design of future systems and contribute to advancements in virtual forklift training.