Assessing Meaningful Visual and Tactile Feedback for Effective Automated Vehicle Takeover by Hearing and Non-hearing Drivers

Abstract

With SAE Level 3 automated vehicles handling most driving tasks, there are still situations when the driver needs to take over. Multimodal displays have been introduced to inform drivers of the need to take over for critical scenarios (e.g., in construction zones) in instructional or informative formats. However, the effects of multimodal displays on takeover performance for drivers with hearing impairments are still unclear. Therefore, the goal of this study was to investigate how signal type (single tactile (T), single visual (V), and visual and tactile combined (VT)), information type (instructional, informative, and baseline), and hearing impairment (hearing-impaired and non-hearing-impaired drivers) affect drivers’ takeover performance. Findings show that signal type significantly influenced reaction and takeover times, with multimodal signals (VT) resulting in faster reactions compared to single modal signals. Additionally, the baseline condition yielded the faster reaction times compared to both instructional and informative formats. Hearing impairment, however, did not significantly affect reaction and takeover times. Findings may inform the development of future vehicle interfaces to assist drivers with hearing impairments.

There are still unexpected situations that require drivers to take over the vehicle in semi-autonomous driving, such as an unforeseen obstacle on the road. This takeover process involves the signal response phase and post-takeover phase to regain vehicle control effectively (Huang & Pitts, 2022). Studies indicate that multimodal displays (combined visual, auditory, and/or tactile displays) result in faster driver reaction times during takeovers compared to single displays (Jeon, 2019).

Multimodal displays have been introduced in instructional and informative formats as takeover requests (e.g., Martinez & Huang, 2024). Instructional displays, such as arrows indicating recommended maneuvers, lead to faster reaction times and improved takeover performance (Van Erp & Van Veen, 2004), reducing cognitive load by guiding attention (Zeeb et al., 2017). On the other hand, informative displays use icons to depict surrounding vehicles and were found to enhance situation awareness (McKerral et al., 2023). However, the effects of informative and instructional formats in single or multimodal signals on takeover performance, considering individual differences, have not been thoroughly compared.

In this context, individuals with hearing impairments face additional challenges in the driving environment due to reliance on auditory cues, highlighting the need for tactile and visual cues. Currently, more than 1.5 billion people have some degree of hearing loss due to factors such as aging populations, exposure to loud noises, and lack of access to care, and this number is projected to reach 2.5 billion by 2050 (World Health Organization, 2024). These challenges include an such as the impaired ability to perceive auditory warning signals (e.g., emergency vehicle sirens; Fostick & Fink, 2021). The introduction of tactile and visual cues presents a viable option to mitigate these risks and enhance road safety for individuals with hearing impairments (Thorslund et al., 2013). Therefore, this study aims to explore how hearing impairments and multimodal displays across various information types might affect takeover performance in automated driving.

The experiment employed a 3 (information type: instructional, informative, and baseline) × 3 (signal type: tactile (T), visual (V), and multimodal (VT)) × 2 (hearing ability: hearing-impaired and non-hearing-impaired) full factorial design. Forty volunteers participated in this study, with twenty participants in each of the hearing groups, driving on a three-lane highway using a medium-fidelity driving simulator while playing a matching game. To simulate hearing impairment, participants in the hearing-impaired group were required to wear noise-canceling headphones and earplugs throughout the experiment. Occasionally, participants needed to take over the vehicle and decide which lane (i.e., either left or right lane) to move into to avoid an approaching car behind on one side, as well as a construction zone in front in the middle lane. A total of three drives were performed with different information types (instructional, informative, and baseline) randomly ordered, with tactile, visual, and multimodal cues deployed twice for each information type. In the instructional scenario, a green arrow pointing left or right was displayed, and a vibration signal moved vertically across their back. Informative signals showed a red car approaching from behind a black car, with corresponding tactile signals indicating the approaching vehicle’s side. Baseline signals vibrated in the middle row of the seat and displayed a red circle visually, prompting drivers to rely on their mirrors.

Takeover performance was measured using reaction and takeover times. Findings revealed that the multimodal display (VT) yielded faster reaction times and takeover times compared to single displays (V or T). Additionally, the baseline had the fastest reaction times compared to informative and instructional signals. There was no main effect of hearing impairment on takeover performance, potentially because the participants were only simulating hearing impairment and were not actually hearing-impaired. The findings may inform the design of human-machine interfaces for automated vehicles and enhance the accessibility for all drivers.