Capturing the Mind: Non-Driving-Related Tasks as a Window into Cognitive Engagement in Automated Driving

Abstract

Supervising automated driving requires visual, manual, and cognitive engagement from the driver. While visual and manual engagement measures are already developed and implemented, cognitive engagement is much more difficult to observe. Considering drivers’ self-regulatory behavior when involved in non-driving-related tasks (NDRTs) during partial automation, the demand and duration of these NDRTs may indicate a certain level of driver cognitive engagement. The study explores a rating measure to partially estimate driver cognitive engagement based on the demand and duration of NDRTs and evaluate the psychometric properties of the NDRT rating. We examined the internal consistency of the measure by computing split-half reliability, tested its sensitivity to factor manipulation, and assessed criterion validity. Our initial findings suggest that the method is reliable and valid, showing satisfactory internal consistency, sensitivity to experimental manipulation, and criterion validity in predicting the takeover performance of supervising vehicle automation.

Objectives

Although eyes-on-the-road and hands-on-the-wheel reflect some degree of driver attentiveness, ensuring the mind is also on the road requires additional effort. An initial step is to quantify driver cognitive engagement in supervising automated driving. This study explores a method to partially estimate driver cognitive engagement based on the demand and duration of non-driving-related tasks (NDRTs).

In an ideal situation, a driver is engaged in supervising partial automation with eyes on the road, hands on the wheel and the mind attentively processing critical vehicle and environment information. The visual, manual, and cognitive aspects are tightly linked together. For example, an attentive driver would be actively scanning the road environment and holding the wheel for potential actions. Similarly, when a driver is not looking at the road, the mind is likely not thinking about driving either. Merat et al. (2019) proposed that drivers engage in both physical control and situation monitoring to be “in the loop.” Physical control manages the vehicle’s basic motions, such as lateral and longitudinal movements. Situation monitoring involves active information processing, requiring drivers to attend to and comprehend the driving situation. Along this line, driver engagement can be reflected in visual behavior (Louw et al., 2016), hand placement (Yuen & Trivedi, 2019), and measures that index cognitive processing. Although visual and hand behaviors reflect certain levels of attentiveness, there are situations when driver cognitive engagement decouples from these behaviors (Mehler, 2020). For example, even when a driver is looking at the road scene, the driver may miss important road information due to cognitive caveats such as inattentional blindness (i.e., look but not see, Saryazdi et al., 2019), subsequent search misses (i.e., see one but miss another, Sall & Feng, 2019), or performing verbal auditory NDRTs, such as talking and listening to music.

This decoupling calls for measures of cognitive engagement. Visual and manual behaviors are more explicit and thus can be assessed from eye or head tracking, motion tracking, or sensing via the wheel (Halin et al., 2021). In contrast, cognitive engagement is more implicit and difficult to measure. One way is to rely on observer ratings (Radwin et al., 2017), as human observers can judge another person’s level of engagement based on action and face cues (Sümer et al., 2021). Considering that drivers tend to be involved in NDRTs during partial automation, and NDRTs differ in demands and duration (Clark et al., 2017; Müller et al., 2021), observing NDRT involvement can shed some light on driver engagement in supervising vehicle automation. Studies suggested that NDRT involvement is influenced by diverse driving conditions, including traveling speed (Seaman et al., 2022), level of automation (Forster et al., 2020), duration of automation use (Forster et al., 2020), and even in-vehicle display settings (Li et al., 2023), likely due to differential cognitive engagement in the driving task under varying conditions. By evaluating NDRTs based on their cognitive demands and duration, it may be possible to indirectly gauge cognitive engagement in partial automation, with the two tasks competing for cognitive processing.

Approach

The data used in the current analyses were collected in two simulated driving studies with a total of 48 undergraduate participants. In each study, 24 participants completed six simulated drives with partial automation during which they were instructed to engage in NDRTs when safe to do so. In Study 1, the engagement was voluntary, and participants could determine whether, when, and in what activities to engage. In Study 2, participants were incentivized to complete anagram puzzles on a phone, but any crash would reduce the incentive. Therefore, they still needed to prioritize driving safety.

For the current analyses, participants’ NDRT involvement was rated based on the cognitive demands of each NDRT and weighted based on the duration as a ratio of the entire drive. To evaluate the psychometric properties (Cook & Beckman, 2006) of the NDRT rating as a cognitive engagement measure, we computed split-half reliability to assess internal consistency, examined sensitivity to factor manipulation, and analyzed its capability to predict takeover performance and workload as a measure of criterion validity. To perform the split-half reliability, six drives in each study were divided into two halves with matching experimental conditions. The sensitivity analysis was based on comparing NDRTs in the two studies in which differential instructions were given. The criterion validity analysis was performed by examining how well the NDRT rating predicts takeover response time and workload as measured using NASA-TLX (Hart, 2006).

Findings

The results suggest NDRT rating as a measure with good psychometric properties. The split-half reliability analysis showed strong correlations between the two halves of the drives in both Study 1 (r = .918, p < .001) and Study 2 (r = .949, p < .001). This finding demonstrates that the measure is internally consistent and that repeated measures within a small time window yield reliable results. To evaluate whether the measure is sensitive to different instruction conditions, an independent t-test was used to compare the NDRT rating in Study 1 (M = 0.697, SD = .933) and Study 2 (M = 2.411, SD = 0.840). The result showed a significant difference in NDRTs between the two studies (i.e., instructions), t(285) = 16.352, p < .001. In addition, regression analyses were performed to assess how NDRT rating can predict takeover reaction time and mental workload during takeover. The rating explained a small (Cohen, 1988) yet significant amount of variance in takeover reaction time (R² = .074, p < .001) and mental workload (R² = .078, p < .001), suggesting that the rating can partially explain the consequence of the level of cognitive engagement. High cognitive engagement in supervising vehicle automation leads to better situation awareness thus better takeover performance and lower workload with the sudden onset of takeover. Therefore, the rating having some predicting value of the takeover performance can reveal how cognitively engaged drivers were.

Takeaways

The study explores NDRT rating based on task demand and duration as a measure of driver cognitive engagement in supervising partial automation. Initial evidence supported the reliability and validity of the NDRT rating that this measure possesses satisfactory internal consistency, sensitivity to experimental manipulation, and criterion validity to predict takeover performance of supervising vehicle automation. Our findings pave the way for a potential window into cognitive engagement in automated driving which tends to be implicit and difficult to measure. The current findings also call for future investigations of other aspects of NDRT involvement, such as how the temporal patterns of task type and individual differences affect NDRT demands. One potential method would be to establish an individual’s baseline for the demands and the patterns of various NDRTs.

Despite the encouraging results from the current analyses, it is important to recognize that NDRT rating may not uncover all aspects of cognitive engagement. For example, mind wandering can absorb cognitive resources in internal non-driving-related mental activities, and the rate of mind wandering increases as the external perceptual demand for driving decreases (Geden et al, 2018). Understanding driver cognitive engagement when using automation requires the consideration of the three-pronged problem—supervising vehicle automation, involvement in NDRTs, and any unrelated internal mental processes all consume the attentional resource. Furthermore, research on real-time indicators of a driver’s likelihood of experiencing inattentional blindness and subsequent search misses remains to be conducted. Starting with a better understanding of NDRTs is just the very beginning.