Investigating Drivers’ Awareness of Automated Vehicle Modes Using the State Diagram Method

Abstract

Driving automation introduces multiple driving modes to maximize the system’s benefits, but drivers must monitor and stay aware of these modes, which can sometimes lead to mode confusion. We modified Degani and Heymann’s state diagram method to assess the mode structure of a hypothetical driving automation system and the likelihood of discrepancies between drivers’ mode awareness and the system’s actual mode. We used the modified method and driving simulation data from participants who weren’t fully informed about all automation modes. The modified method visualized all possible combinations of automation modes and drivers’ mode awareness, highlighting where they diverge and estimating the frequency of the divergence. The diagram identified areas where human-machine interface (HMI) design can help drivers maintain accurate mode awareness. The modified state diagram method provides actionable insights for designing HMIs to reduce mode confusion and can be used to develop computational models of automation mode structures.

Background

As driving automation advances, multiple modes are introduced to increase system benefits. Drivers can select the mode that best meets their needs, but multiple modes demand drivers to be aware of the automation’s mode structure. However, maintaining accurate mode awareness can sometimes fail and lead to mode confusion. Mode confusion occurs when the human operator’s awareness of automation’s current mode and the actual mode differ (Sarter & Woods, 1995). Drivers face various challenges in maintaining accurate mode awareness. The current automated vehicles feature 4 to 5 modes, such as cruise control (CC) and adaptive cruise control (ACC). Between these modes, 50 to 72 transitions are possible (Janssen et al., 2019) that need monitoring. Mode transitions in SAE Levels 2 to 3 automation occur frequently (Drüke et al., 2018; Hipp et al., 2018), some initiated by automation without warning. Automation features often include similar modes, with transitions between them frequently going unnoticed by drivers, such as transitioning from ACC to CC (Boos et al., 2020; Horiguchi et al., 2007). Despite these challenges, drivers are not offered sufficient training opportunities to learn about automation features (McDonald et al., 2018).

Degani and Heymann (2002) proposed the state diagram method to analyze complex mode structures and identify root causes of mode confusion. It leverages the formal method, a computer science approach for specifying a complex system (e.g., software). This method depicts a system as comprising various states, with transitions between states defined using mathematical formulas or formal logic (Taylor & Darrah, 2005). Degani and Heymann (2002) suggested developing separate formal models for automation’s mode structure (automation model) and operator’s mode awareness (operator model). These two models may not always align because state transitions in the automation model occur based on its design. In contrast, those in the operator model occur based on operator instruction and knowledge. Integrating the automation and operator models can reveal mode transitions where the user awareness and the actual mode don’t align. This allows for improvements in the automation’s mode structure by removing confusing transitions. Another benefit of a formal model is that it can simulate a system, enabling the estimation of the likelihood of discrepancies between drivers’ mode awareness and the actual mode. However, this feature is currently absent in Degani and Heymann’s (2002) state diagram approach. The current work extends their approach to estimate the likelihood of mode confusion and identify methods to mitigate mode confusion.

Method

Hypothetical Advanced Driver Assistance System

We analyzed the mode structure of a hypothetical SAE Level 2 Advanced Driver Assistance System (ADAS) from our previous study (Lee et al., 2023). This ADAS had seven modes: manual (0), automation-ready (auto-ready, 1), lateral override (2), longitudinal override (3), Level 2 automated driving (Level 2, 4), amber alert (5), and red alert (6).

In manual mode, participants drove the vehicle. The auto-ready mode was activated when they pressed the automated driving button on the steering wheel while in manual mode (Figure 1a). Participants still had to drive, but the ADAS actively assessed the road and switched to Level 2 mode when its operating conditions were met. In Level 2 mode, the ADAS handled both longitudinal and lateral vehicle control. Lateral override was activated when participants steered to take lateral control while in Level 2 mode (hence the term “override” in its name). This mode was similar to ACC in that the ADAS managed longitudinal control, but the participants performed lateral control. Conversely, the longitudinal override was activated when they accelerated to perform longitudinal control, and the ADAS managed lateral control, similar to the lane-keeping assist system. The ADAS automatically returned to Level 2 mode when no control input was registered after an override. The mode changed back to auto-ready mode when they braked in Level 2 mode. The ADAS issued an amber alert when its confidence in driving dropped below a threshold, warning participants to monitor its behavior closely while maintaining the Level 2 mode functionality. The ADAS also issued a red alert when it could not handle a road situation. If the participants didn’t take control, it stayed in Level 2 mode for 5 s before switching to the auto-ready mode.

Figure 1.

The HMI of the hypothetical ADAS during a red alert: (a) the steering wheel light signaled red and amber alerts and whether the ADAS performed lateral control and (b) the set speed symbol on the dashboard signaled whether it performed longitudinal control, and the steering wheel symbol communicated the same information as the steering light.

These modes were communicated through the vehicle HMI. Four HMI features changed throughout the driving simulation depending on the mode: Steering wheel LED light (steering light), set speed symbol, steering wheel symbol (steering symbol), and text message (message). The luminance and color of these features differed in each mode. The steering light (Figure 1a) signaled whether the ADAS performed lateral vehicle control (green) or not (turned off). It also signaled amber and red alerts (amber, red). The set speed symbol (Figure 1b) indicated whether the ADAS performed longitudinal control (green) or not (gray and blue). The blue set speed symbol indicated the ADAS was in longitudinal override mode. The steering wheel symbol communicated the same information as the steering light. The dashboard messaged “Take control of your vehicle” when amber or red alerts were issued. Further descriptions of the HMI features are provided in Figure 3 for easier visualization and comparison between the modes.

Participants were not fully instructed on all ADAS modes due to the scope of the previous study (Lee et al., 2023). They were not told that the auto-ready mode existed but that Level 2 mode would activate once they pressed the automated driving button in manual mode and Level 2 mode conditions were met. Moreover, the instructions referred to both lateral and longitudinal override modes as the temporary override mode, allowing the driver to change the vehicle’s course temporarily while in Level 2 mode.

State Diagram Development

Figure 2 shows the modified state diagram method procedure. This procedure used data and instruction materials from Lee et al. (2023).

Figure 2.

The modified procedure identifies the root causes of mode confusion by integrating the ADAS and driver models into the composite model.

An ADAS model, a driver model, and a composite model were generated. The ADAS model was developed using the ADAS mode information from the driving simulator data and the discussion with researchers who designed the hypothetical ADAS. The resulting ADAS model was a rule-based model, defining specific conditions for mode transitions (ground truth; IF condition A, THEN transition to mode 1). The driver model was developed based on participant instruction and the qualitative assessment of participant behavior in the study. We reviewed the study’s video recordings to identify instances of drivers expressing mode confusion, defined by one of the following criteria. The first criterion was when participants showed confusion through facial or bodily expressions, later confirmed by their verbal comments or by the experiment moderator reminding them of the ADAS’s mode. The second criterion was when participants removed their hands or feet from the control devices despite the ADAS not being in Level 2 mode. The instances meeting these criteria were then reviewed and validated through discussions in a meeting with five researchers. The resulting driver model was a Markov transition matrix, representing participants’ awareness of mode transitions and their probabilities.

Integrating these models, the composite model uses a state diagram to present all potential combinations of ground truth modes and driver’s mode awareness as states, along with state transition probabilities and the HMI information presented for each state. From this diagram, the blocking and error states where drivers’ mode awareness diverges from ground truth can be identified, and their likelihood can be estimated by examining the frequencies of the transitions leading to these states. Blocking states occur with unexpected transitions drivers are unacquainted with, while error states arise when drivers are unaware or falsely believe the ADAS is in an undesired mode. We assumed manual mode as undesired because the previous study motivated participants to use the ADAS for as long as possible.

The composite model identifies the modes or transitions where the driver’s awareness diverges from the ground truth mode but does not fully list the information sources available to drivers for maintaining mode awareness. Therefore, we specified the HMI information shared in each state in the modified composite model.

Result

The composite model identified two blocking states and an error state among the 17 possible transitions across the seven modes (Figure 3). One blocking state (0A), accounting for about 2.3% of all transitions, was associated with the automated driving button. From manual mode, pressing this button didn’t immediately activate Level 2 mode but transitioned the mode to auto-ready mode. Level 2 mode was activated automatically only when the ADAS was in an automation-enabled area and exceeding 17 mph while in auto-ready mode. However, some drivers inaccurately believed that pressing the automated driving button would directly activate Level 2 mode, failing to anticipate the transition from manual mode to auto-ready mode (from 0A and 1C and vice versa in Figure 3).

Figure 3.

The composite model showing all potential combinations of ground truth modes and driver’s mode awareness, each represented by a table showing the ground truth ADAS mode (ADAS model), the driver’s awareness (driver model), and the information provided by the vehicle HMI.

Moreover, drivers were instructed to respond to the red alert only by braking. This caused another blocking state (6E) comprising 3.2% of the transitions, in which drivers were unaware that they could also steer and activate lateral override after red alert (from 6E to 2B). The error state (1A), accounting for about 27% of all transitions, was associated with braking during Level 2 mode. Some drivers thought that braking would switch to manual mode, but it led to auto-ready mode (transitions leading to 1A from 4C, 5D, and 6E).

Discussion

This study evaluated a hypothetical ADAS, identifying states where drivers’ mode awareness diverges from the ground truth. Two blocking states were identified where participants unexpectedly transitioned the ADAS to auto-ready and lateral override modes. Moreover, participants’ inaccurate awareness of the transition initiated by braking caused an error state, where they falsely believed the ADAS was in manual mode while it was in auto-ready mode.

In turn, these insights can be used to evaluate the driver manual or the vehicle HMI design to find solutions for mitigating mode confusion. Our approach to estimating the frequency of ground truth-mode awareness divergence and presenting the information shared with the driver offers actionable insights into how the driver manual or HMI design can be improved. For example, to prevent drivers from falsely believing they are in manual mode when in auto-ready mode, the dashboard screen for auto-ready mode can be redesigned to clearly indicate the possibility of the Level 2 mode activation. Alternatively, it can be redesigned to present a visually distinct screen compared to manual mode, other than presenting gray set speed and steering symbols.

One question requiring further investigation is how to define a mode. Sarter and Woods (1995) suggested that modes are selected by automation users based on what they think is appropriate for handling the situation at hand. According to this definition, amber and red alerts are not modes and should be excluded from the state diagram because transitions to these states are initiated by automation. This would significantly impact the state diagram and the solutions generated from it. In our example, the blocking state occurring during transitions from red alert to lateral override mode would be excluded. Future work is needed to determine what should be defined and included as a mode in the state diagram.

Our modified method offers more actionable information for designing driver manuals and HMIs to mitigate mode confusion (e.g., how and which states to improve). Furthermore, the modified state diagram can be converted into a Markov transition matrix specifying the likelihood of transitions to states where driver awareness diverges from ground truth. Future studies can utilize this matrix to develop a computational model and test the automation mode structure of interest.