Abstract
This study investigates the impact of a supportive system on takeover transitions in conditionally automated driving (SAE Level 3). The supportive system is designed to direct drivers’ attention toward potential hazards in the environment when a takeover request occurs. The study comprises two components: (a) identifying various types of potential hazards using naturalistic driving data, and (b) conducting a driving simulator study to develop and assess a gaze guidance system based on the N-SEEV model of visual attention. Results indicate that drivers using a highly salient attention guidance system were less likely to collide with a secondary hazard during takeover transitions. This suggests that gaze guidance support is an effective approach for assisting drivers during takeover transitions.
Introduction
The development of autonomous vehicles represents a significant leap in transportation technology. The appropriate use of autonomous vehicles can improve traffic safety and efficiency by reducing human errors and failures (Kalra & Paddock, 2016; National Highway Traffic Safety Administration [NHTSA], 2017). Autonomous vehicles can also provide a convenient experience to drivers by decreasing workload and enabling engagement in non-driving-related tasks (NDRT) during commutes (Du et al., 2020b; Zhou et al., 2020).
According to the Society of Automotive Engineers (SAE) classification, vehicles classified as level 3 or higher possess autonomous driving capabilities. Level 3 autonomous vehicles are capable of executing driving tasks and monitoring the environment. However, they require the driver to assume control when prompted by the system (SAE International, 2014). While operating autonomously, drivers are disengaged from active vehicle control, leading to significant challenges in quickly reassuming control when necessary (Ayoub et al., 2019; Du et al., 2020a; Eriksson & Stanton, 2017; Gold et al. 2016).
Moreover, previous research indicates that difficulties or diminished performance during takeover transitions are typically not caused by drivers’ inability to detect the event triggering the Takeover Request (TOR) but rather due to their struggles with quickly assessing the surrounding traffic situation (Zeeb et al., 2016). Drivers may focus their attention exclusively on the TOR-triggering event and overlook other critical information, a phenomenon commonly known as “tunnel vision” (Briggs et al., 2016; Ringer et al., 2016; Williams, 1988). Objects near the ego vehicle may become hazards depending on the maneuvers drivers choose to respond to the TOR-triggering event. For instance, a TOR may be initiated by the breakdown of a leading vehicle, prompting the driver to change lanes to avoid a rear-end collision. However, if there is an unnoticed vehicle in the ego vehicle’s blind spot, the lane-changing maneuver could result in a collision.
This study, therefore, aims to (a) identify potential hazards that co-occur during takeover transitions and (b) design and evaluate a gaze guidance system to support drivers’ noticing of potential hazards during takeovers.
Study 1: Identification of Potential Hazards
Sampling and Categorization
We identified potential hazards that could emerge during takeover transitions by mining the IVBSS database. IVBSS, Integrated Vehicle-Based Safety Systems program, is a study led by the University of Michigan Transportation Research Institute to test integrated crash avoidance system that includes the for- ward crash warning (FCW), lateral drift warning (LDW), curve speed warning (CSW), and lane change/merge warning (LCM) (Sayer et al., 2011). Sixteen passenger sedans were equipped with the system sensor and were used by 108 volunteer drivers recruited in southeastern Michigan. The data collected contained 213,309 miles, 22,657 trips, and 6,164 hr of driving.
A researcher randomly sampled videos for each type of warning from the IVBSS database. These videos were viewed to identify common themes/situations that could be considered (additional) potential hazards besides the warning-triggering event. These common themes/situations were discussed among the research group to determine. Once a consensus was achieved, a coding rule was developed, considering the type of warning and the possible maneuvers a driver might perform during the warning.
After that, approximately 200 videos were randomly sampled from the IVBSS database for each warning type, each lasting seconds. The following selection criteria were applied: (a) a driver had to have at least 50 miles of driving data, (b) a driver was traveling at a speed of at least 25 mph during the video, and (c) the duration of continuous driving in the video was at least 30 s. The query resulted in 194 videos for FCW, 199 videos for LCM, 193 videos for LDW, and 211 videos for CSW, resulting in a total of 797 videos.
Results: Potential Hazard Scenarios
A total of 10 potential hazard scenarios were derived from the four types of warnings. Notably, the hazards of interest are not the direct causes of the warnings. Rather, they are additional hazards present in the environment at the time of the warning.
A FCW is issued to alert the driver of a potential rear-end collision between the ego vehicle and the leading vehicle. When an FCW is triggered, we identified two possible hazards: Car in Blind Spot (FCW-BS)—a vehicle located in the blind spot of the ego vehicle (see Figure 1), and Car in the Adjacent Lane (FCW-AL)—a vehicle situated in the lane adjacent to the ego vehicle.
Illustration of FCW-BS. Ego vehicle (red) issues a forward collision warning (FCW) due to the lead vehicle (white) and there is a vehicle in the blind spot (blue).
A LCM warns drivers of possible unsafe lateral maneuvers as the ego vehicle changes into an adjacent occupied lane. When a LCM is triggered, we identified three potential hazards: Car in Blind Spot After Changing Lanes (LCM-BSCL)—a vehicle in the blind spot after the lane change maneuver, Car in Adjacent Lane After Changing Lanes (LCM-ALCL)—a vehicle in the adjacent lane within close proximity to the ego vehicle, and Car in Blind Spot in the Departed Lane (LCM-BSDL)–a vehicle in the originally departed lane within close proximity to the ego vehicle.
A LDW is issued as the ego vehicle is drifting toward the other lane. When a LDW is triggered, we identified three potential hazards: Car in the Opposite Direction (LDW-OD)—a vehicle traveling in the opposite direction. Car in the Adjacent Lane (LDW-AL)—a vehicle traveling in the lane where the ego vehicle drifted toward within close proximity, and Car in Blind Spot (LDW-BS)—a vehicle traveling in the lane where the ego vehicle drifted toward in the blind spot.
A CSW is issued as the ego vehicle over speeds when entering a curve. When a CSW is triggered, we identified two potential hazards: Car in the Opposite Direction (CSW-OD)—a vehicle traveling in the opposite direction, and Slower Lead Vehicle (CSW-SLV)—a leading vehicle traveling in the same direction with a slower speed.
Table 1 shows the descriptive statistics of the video categorization. For detailed descriptions and graphic illustrations, please refer to Han et al. (2023).
Categorization Result from Video Samples.
Study 2: Development of a Gaze Guidance System
Design of a Gaze Guidance System
In this study, we designed and evaluated a gaze guidance system to support the noticing of potential hazards. The gaze guidance system was motivated by the N-SEEV (Notice-Salience, Effort, Expectancy, Value) model of selective visual attention, with a particular emphasis on the salience factor (Wickens, 2015). The system was designed to draw attention to important, safety-critical information for drivers to focus on when they resume control from automation. Different levels of salience designs were evaluated. After identifying potential hazards in Study 1 and considering practical aspects for implementation in a driving simulator, the FCW-BS scenario was selected for further investigation in Study 2 (see Figure 1).
Participant
A total of N = 12 participants (average age = 25.8 years, SD = 4.4 years, six females, six males) with normal or corrected- to-normal vision participated in the experiment. Each participant received a payment of $30 for their participation. The study was approved by the University of Michigan Institutional Review Board.
Experimental Design
The experiment used a one-way within-subjects design with salience level (high salience, low salience, and control) as the independent variable. For both high and low salience conditions, the side mirror was highlighted with a red rectangular bounding box when a potential hazard located near the ego vehicle was about to enter the blind spot of the ego vehicle at the moment of TOR (Figure 2). The difference between high and low salience conditions was the flashing pattern of the bounding box. In the high silence condition, the red box would flash 20 times over 4 s (5 hz), whereas in the low salience condition, the red bounding box would appear for 4 s without flashing.
Illustration of gaze guidance system from top to bottom: control condition, low salience guidance.
Each participant experienced all three salience conditions (high salience, low salience, and control). The order of the three conditions was balanced using a balanced Latin square design. Each participant encountered three takeover events in the experiment, due to either a construction site ahead, or a police officer on the road shoulder, or a stationary bus ahead (Figure 3). Additionally, at the moment of TOR, a car behind the ego vehicle was about to enter its blind spot, indicated as the blue vehicle Figures 1 and 2. If the ego vehicle attempted to change lanes without noticing the blue vehicle, a rear-end crash/near crash would occur; If the ego vehicle waited for the blue vehicle to pass before switching lanes, no crash would occur.
Illustration of takeover events: (a) Construction ahead, (b) Police officer on shoulder, (c) Stationary bus ahead.
We measured several dependent variables, including the occurrence of crash/near crash, the time taken to notice potential hazards, and subjective evaluations (Table 2).
Subjective Measures of Perceived Readiness, Level of Engagement, Perceived Usefulness, and Ease of Use for the Gaze Guidance System.
The time to notice the potential hazard vehicle was calculated as the time between the issuance of TOR and the first fixation on the side mirror (Figure 4).
Illustration of the first fixation on the side mirror.
Additionally, we measured participants’ perceived takeover readiness, their level of engagement in the NDRT before TOR, and their perceived usefulness and ease of use for the gaze guidance system. At the end of the experiment, participants were asked to rank their preference for the three designs (i.e., high salience, low salience, and control).
Procedure
Upon arrival, participants provided informed consent and filled out a demographic survey followed by an introduction that described the content of the experiment. Next, participants received training sessions to get familiar with the driving simulator and the Tetris game. During the training, participants practiced how to drive, change lanes, and engage/disengage in the automated driving mode. They were also presented with visual and auditory alerts for takeover requests. Participants were then asked to drive the simulator until they were familiar with handling the simulator controls. After that, participants played the Tetris game to get familiarized with it. After the training, participants were asked to wear the eye tracker.
The main experiment began with the command to activate the automated driving mode. After that, there was an NDRT phase where participants were asked to play the Tetris game. Participants were informed that there was no need to monitor the environment when the AV was in automated driving mode. Once a TOR was issued, participants were required to take over control of the vehicle immediately. Participants were instructed to comply with all the traffic laws when they drove manually. They were informed that the speed limit was 35 mph. Participants could hand back the control to the AV after they negotiated the driving situation. When the AV was re-engaged, participants filled out a subjective questionnaire. At the end of the experiment, participants ranked their preferences for the three designs.
Results
Table 3 shows the occurrences of crashes/near-crashes and the noticing time as participants encountered the three events. Despite the use of a Latin square design, we noticed significant learning effects as participants experienced more takeover events. Therefore, we focused the data analysis on the first event, event 1.
Occurrence of Crash/Near Crash and Noticing Time.
Note. Crash: 1 = Yes, 0 = No, NC = Near Crash.
Occurrence of Crashes/Near Crashes
We conducted a χ2 test to compare the likelihood of crashes/near crashes occurrence between the three salience conditions. The result shows a significant difference, χ2(2) = 6.0, p = 0.05, that there was less likelihood of crashes/near crashes with the high salience design, compared to the low salience design and the control condition.
Noticing of Potential Hazard
We conducted a one-way analysis of variance (ANOVA) to compare the noticing time between the three conditions, which showed a non-significant difference (F [2, 6] = 0.98, p = 0.428).
Subjective Evaluation and Ranking
Table 4 tabulates the mean and SD of perceived readiness, engagement in NDRT, perceived usefulness, and perceived ease of use. There seemed to be trends that the high salience engenders higher perceived readiness, usefulness, and ease of use. However, one-way ANOVA revealed non-significant differences in perceived readiness (F [2, 9] = 2.29, p = 0.157), engagement in NDRT (F [2, 9] = 0.00, p = 0.807), perceived usefulness (F [1, 6] = 1.34, p = .290), and perceived ease of use (F [1, 6] = 2.23, p = 0.122)
Mean and Standard Deviation (SD) of Perceived Readiness, Engagement in NDRT, Perceived Usefulness, and Perceived Ease of Use.
Figure 5 shows the ranking of the three salience conditions (1st—most preferred, 3rd—least preferred). A Friedman’s test shows a significant difference between the ranks (χ2(2) = 18.5, p < 0.001). The high salience design was preferred the most, followed by the low salience design and the control condition.
Illustration of takeover events.
Discussion
Despite the extremely limited number of participants, our study yielded noteworthy findings. The high salience design significantly reduced the likelihood of crashes, with no incidents reported among the four participants in this group. In contrast, three crashes were observed in both the low salience and control conditions. To our surprise, there does not seem to be a difference between the low salience and control conditions. Notably, three out of four participants in the low salience group failed to check the side mirror, as indicated by a “-” sign in Table 3. This suggests the necessity of an attention-grabbing mechanism to redirect the driver’s attention to the potential hazards. The participants’ evaluation indicated a trend in favor of the high salience condition, with higher perceived performance, usefulness, and ease of use compared to the low salience condition. This trend was supported by the significant results from the ranking analysis (χ2(2) = 18.5, p < .001), where 10 out of 12 participants rated the high salience design as the most preferable option.
The study should be reviewed in light of the following limitations. Firstly, the sample size was notably small, which may restrict the generalizability of the findings. To validate these results, further studies with larger sample sizes are recommended. Secondly, the design of the gaze guidance system was primarily influenced by the salience factor within the N-SEEV model of selective visual attention. Future research could benefit from exploring system designs that incorporate other factors from this model.
Conclusion
This proof-of-concept study lays the foundation for future research and development in the field of conditional autonomous vehicles, where there are additional hazards other than the hazard that triggers the TOR. Our study aimed to demonstrate the potential of a gaze guidance system in enhancing takeover readiness and performance. Due to significant learning effects, our analysis focused solely on the first takeover event, involving four participants in the high salience condition, four in the low salience condition, and four in the control condition. Our results have shown significant differences in the occurrence of crashes/near crashes and the preference in design. The high salience design resulted in fewer crashes and was the most preferred design.
Footnotes
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This material is based upon work supported by the AAA Foundation for Traffic Safety.
