Sage Journals: Discover world-class research

Abstract

Objective

Using brain haemodynamic responses to measure perceived risk from traffic complexity during automated driving.

Background

Although well-established during manual driving, the effects of driver risk perception during automated driving remain unknown. The use of fNIRS in this paper for assessing drivers’ states posits it could become a novel method for measuring risk perception.

Methods

Twenty-three volunteers participated in an empirical driving simulator experiment with automated driving capability. Driving conditions involved suburban and urban scenarios with varying levels of traffic complexity, culminating in an unexpected hazardous event. Perceived risk was measured via fNIRS within the prefrontal cortical haemoglobin oxygenation and from self-reports.

Results

Prefrontal cortical haemoglobin oxygenation levels significantly increased, following self-reported perceived risk and traffic complexity, particularly during the hazardous scenario.

Conclusion

This paper has demonstrated that fNIRS is a valuable research tool for measuring variations in perceived risk from traffic complexity during highly automated driving. Even though the responsibility over the driving task is delegated to the automated system and dispositional trust is high, drivers perceive moderate risk when traffic complexity builds up gradually, reflected in a corresponding significant increase in blood oxygenation levels, with both subjective (self-reports) and objective (fNIRS) increasing further during the hazardous scenario.

Application

Little is known regarding the effects of drivers’ risk perception with automated driving. Building upon our experimental findings, future work can use fNIRS to investigate the mental processes for risk assessment and the effects of perceived risk on driving behaviours to promote the safe adoption of automated driving technology.

Keywords

aggressive and risky driving autonomous driving cognitive neuroscience human-automation interaction risk assessment

INTRODUCTION

Assessing drivers' functional state is essential to ensure the safe adoption of automated driving technology (Alrefaie et al., 2019; Lohani et al., 2019; Perello-March et al., 2021; Wörle et al., 2019). Driver monitoring systems (DMS) will soon be a mandatory safety feature for new production vehicles in Europe (EuropeanCouncil, 2019) and the US (NHTSA, 2022).

Current literature mainly relies on gaze-behaviour and well-established peripheral physiology metrics such as heart rate variability, respiration, blood pressure, and skin conductance (Dong et al., 2011; Lohani et al., 2019; Melnicuk et al., 2021). However, driving and supervising an automated system are primarily cognitive and executive tasks. Much relevant data is overlooked when neurophysiology and neural activity are not measured. Electroencephalography (EEG) is the most well-established neuroimaging technique in driving research (Lohani et al., 2019; Seet et al., 2020; Solís-Marcos et al., 2017, 2018), yet the use of functional near-infrared spectroscopy (fNIRS) in driving research has been increasing over the recent years (Balters et al., 2021). Current fNIRS devices are portable, wearable, lightweight kits, robust to movement artefacts, and allow flexible configurations. In addition, fNIRS provides good spatial resolution but slower temporal resolution than EEG (Balters et al., 2021), making fNIRS an ideal technique for naturalistic research.

Driving research using fNIRS has primarily focused on measuring mental workload and fatigue from manual driving (Foy & Chapman, 2018; Z. Li et al., 2009; Lin et al., 2020; Lohani et al., 2019), but little work has been conducted in the context of driving automation. Only a few studies have used fNIRS to explore constructs such as trust in automation in a driving simulator (Perello-March et al., 2023), where the authors found lower prefrontal activation during hazardous events for participants trusting the automated vehicle compared to those distrusting. Another driving simulator study (Balters et al., 2017) indicated that drivers' prefrontal activity tends to decrease with continuous exposure to automated driving, a phenomenon described as habituation. The habituation to automation increases with higher levels of automation. Relatedly, Sibi et al. (2016) observed a similar decrease in prefrontal activity with automation engaged compared to when driving manually.

These studies suggest that reliable driving automation decrease cortical activity. In short, relegating drivers to a mere monitoring role of a ‘reliable’ system leads to underload and a lack of cognitive capacity to perform an optimal take-over of control – also known as being out-of-the-loop (Merat et al., 2019). This phenomenon could be attributed to situation awareness worsening as the automated system assumes greater control of the driving task (Endsley, 2017). Poor situation awareness and reduced monitoring behaviours due to the shared control between the driver and the automation may put drivers at risk when unexpected take-over requests are issued, especially in conditionally automated driving (SAE Level 3) (SAE International, 2021).

We suspect this poor situation awareness is related to a lower perception of risk resulting from the shared control of the driving task. A lack of risk perception from automated driving is not trivial since it can lead to overtrust and automation misuse (Kundinger et al., 2019; Parasuraman & Riley, 1997), which may compromise safety. Indeed, age-related risky behaviours have previously been explored in traffic psychology research with fNIRS (Foy et al., 2016). The authors found that reduced prefrontal cortex activity during several simulated driving tasks was associated with lower risk perception in young males compared to females and older drivers. The authors attribute these results to a lack of prefrontal maturation in younger male drivers.

Extensive traffic behaviour and psychology work has explored risky driving and driver hazard perception from personality traits (Du et al., 2020; Jonah, 1997). In short, driver hazard perception has been defined as drivers’ situation awareness for potentially dangerous incidents in the traffic environment. That is the ability to detect dangerous traffic situations (Horswill & McKenna, 2004). Effective hazard perception can be considered a central executive task based on a dynamic mental model of the driving environment used for actively searching dangerous situations. A mentally effortful and proactive process that requires working memory and attentional resources (Horswill & McKenna, 2004).

Another related concept that is often confounded with hazard perception is risk perception. It is the subjective evaluation of how well drivers think they – or the driving automation – can handle the situation and apply an appropriate action (Borowsky & Oron-Gilad, 2013). Risk perception has been found to have two major components: (1) the likelihood of a crash and (2) the severity outcomes of a crash (Borowsky & Oron-Gilad, 2013; Riley, 1996). To summarise, hazard perception is the skill to detect hazards in real-time, whilst risk perception is the evaluation of the chances of being involved in a crash in a certain situation. Both have strong ties to situation awareness and trust in automation (Endsley, 1995; Muir, 1994).

In a video-based driving hazard detection and evaluation task experiment, Borowsky & Oron-Gilad (2013) found that drivers would put more weight on the likelihood of a crash in real-time driving. On the other hand, they would pay more attention to the severity of the outcome of a crash when required to evaluate risk in hindsight. The authors argue that under the time pressure of driving, drivers usually focus on preventing the crash (i.e., the likelihood of the crash, the first component of risk perception) rather than thinking of the severity outcomes of the crash (i.e., the second component of risk perception). However, whether these findings are transferable to highly automated driving (HAD, SAE Level 4) remains unclear, where drivers are released from the responsibility and time constraints of real-time driving. To put it in other words, are these components of risk perception equally manifested during HAD? Or instead, drivers remain out of the loop since they are not in control?

Taking the notion that hazard perception is a mentally effortful proactive process involving working memory and attentional resources (Horswill & McKenna, 2004) and that hazard perception is a necessary condition for risk perception to exist (Borowsky & Oron-Gilad, 2013), we considered that cortical neurophysiology could be an adequate research tool to investigate this phenomenon. In particular, cortical haemoglobin oxygenation levels could indicate the underlying central executive cognitive processes involved in real-time risk perception assessments. Previous work using fNIRS has evidenced that cortical haemoglobin oxygenation levels indicate different levels of trust during highly automated driving (Perello-March et al., 2023). Hence, since risk perception is a major factor affecting trust (Lee & Moray, 1992; Muir, 1994; Riley, 1996), cortical haemoglobin concentrations could also be used to measure risk perception in this context.

Borowsky & Oron-Gilad (2013) considered several environmental characteristics, such as the nature of the driving environment – for example, urban or residential, as well as traffic complexity as the combination of environmental features – for example, traffic flow and volume and lane changes among other road users (Teh et al., 2014) to be hazard instigator types, and have been found to increase drivers’ stress levels (Foy & Chapman, 2018; Healey & Picard, 2005; Perello-March et al., 2021). These hazard instigator types can appear in different states of progression – that is, materialised (require an evasive response), hidden unmaterialised (obscured by other road objects but not require a response), and potential unmaterialised (visible but not require response). Thus, we have conducted a high-fidelity driving simulator study with two types of hazard instigators:

(1) Potential unmaterialised hazards across suburban and urban driving conditions with moderate levels of traffic complexity slowly building up.

(2) A materialised hazard in a quickly escalating driving scenario requiring an evasive manoeuvre.

We expect cortical prefrontal oxygenation levels to increase if drivers actively search for potential hazards and evaluate the likelihood and severity of the outcomes of a potential crash. On the contrary, low perception of risk should result in a decrement in prefrontal cortical activity. Based on the findings described in the literature review, we hypothesise:

Hypothesis 1: Prefrontal haemoglobin oxygenation levels during suburban and urban driving conditions with moderate levels of traffic complexity and potential un-materialised hazards will not differ from baseline resting and a recovery period due to a lack of situation awareness and perceived risk.

Hypothesis 2: The materialised hazard in a quickly escalating driving scenario requiring an evasive manoeuvre will produce variations in prefrontal haemoglobin oxygenation compared to the baseline resting and recovery period due to perceived risk.

METHOD

Participants

A convenience sample of twenty-three volunteers was recruited to participate in this experiment. Three participants were excluded from the analysis as they dropped out from the experiment due to motion sickness, with the data of 20 participants analysed (10 female, M_age = 24.60, SD = 3.91). All had held a UK-EU driving license for an average of 5.30 years (SD = 4.18) and an average driving experience of 6780 miles/year (SD = 6140.08). Participants were recruited from the University of Warwick (UK), including undergraduate and postgraduate students and professionals. Recruitment and data collection procedures received approval from the University of Warwick’s Biomedical and Scientific Research Ethics Committee. Participants were free to withdraw at any point and did not receive compensation.

Driving Simulator

The experiment took place in the driver-in-the-loop 3xD driving simulator at WMG, the University of Warwick. The 3xD (Figure 1) is a fixed-base high-fidelity driving simulator with a whole-body Range Rover Evoque and 8 projectors generating a 360° image projected into a cylindrical screen 8 metres in diameter and 3 meters in height. The simulated driving automation is capable of lateral and longitudinal control, adapting to speed limits, queuing leading vehicles, maintaining safe distances, emergency braking, and overtaking slower/stopped vehicles for predefined use cases. The road environment sound and motion vibration are played stereo via 2 × 20 W speakers.

Figure 1.

Snapshot of the driving simulator and the virtual environment projected on a curved screen.

Functional Near-Infrared Spectroscopy

Neurophysiological data was obtained from the prefrontal cortex with a NIRSport CW-NIRS device (NIRx Medical Technologies LLC, USA) (Figure 2), using NIRStar acquisition software (CA, USA; version 15.0). NIRSport is a noninvasive wearable device consisting of eight sources and seven detectors sampling at a frequency of 7.8125 Hz. The sources simultaneously emit infrared signals of two distinct wavelengths, 760 nm and 850 nm, allowing quantification of oxygenated haemoglobin (HbO), deoxygenated haemoglobin (HbR), and total haemoglobin (HbT = HbO + HbR). Both chromophores can be differentiated when light attenuation is measured at two or more wavelengths due to their differential absorption spectra in the near-infrared spectrum (600–950 nm).

Figure 2.

Channels montage and representation of the whole setup device – in our case, the hardware was placed behind the passenger seat.

Plastic spacers located at a distance of 3 cm between each source and detector pair constitute a recording channel, thus resulting in 22 recording channels. Channels were mounted within the Brodmann areas coordinate space for consistency across head size variation using the fNIRS Optodes’ Location Decider (fOLD), which is a toolbox for probe arrangement guided by brain regions of interest. The toolbox automatically decides optodes positions based on 10–10 and 10–5 systems according to a set of brain regions of interest (Zimeo Morais et al., 2018). These coordinates allow subsets of fNIRS channels to directly measure particular regions of interest (ROIs) (Table 1)

TABLE 1.

List of Channels and Regions of Interest

Source	Detector	Channel	Brodmann Area
1	1	1	8 right
1	3	2	-
2	1	3	8 right
2	2	4	8 left
2	4	5	8 left
3	2	6	8 left
3	5	7	44 left
4	1	8	8 right
4	3	9	9 right/46 right
4	4	10	9 left/9 right
4	6	11	9 right
5	2	12	-
5	4	13	9 left
5	5	14	44 left/45 left
5	7	15	10 left
6	3	16	46 right
6	6	17	10 right/46 right
7	4	18	9 left
7	6	19	10 right
7	7	20	10 left
8	5	21	45 left
8	7	22	45 left/46 left

Questionnaires

Subjective measures included a bespoke risk perception questionnaire comprising two items which were asked after completing the trial:

(1) Did you feel any sensation of risk or threat from the whole scenario?

(2) Did you feel any sensation of risk or threat from the traffic accident at the end?

These were rated on a Likert scale ranging from 1 (not at all) to 7 (extremely).

Current validated measures for hazard perception are image/video-based tests in which participants are asked to detect promptly potential hazards from several road scenarios (Joanne et al., 2010; Malone & Brünken, 2016). Hazard perception performance is measured by latency (response time) and accuracy (success/failure to detect). However, to the authors’ knowledge, no existing validated self-reported tools for risk perception assessment in the driving context exist.

Li et al. (2019) used the scale from Rajaonah et al. (2008) to assess risk perception associated with trusting in automation. Even though both studies reported significant effects on risk perception, we did not use this scale because it needs to be validated and measures perceived situational and relational risk. In our experiment, we were interested in comparing perceived situational risk with the hazardous event during the automated driving scenario. Thus, this would have implied reporting perceived risk at the end of each condition, which we considered was contraindicated due to our continuous driving experimental design and as the hazardous event occurred immediately after the automated driving conditions. Stopping the scenario immediately before the Driving Hazard event could have affected the realism of the scenario and any neurophysiological reactions.

In addition, the Trust in Automated Systems Scale (Jian et al., 2000) was included to evaluate the perceived risk’s impact in building trust in automation. The scale was rated before and immediately after the trial was completed.

Experimental Procedure

Upon arrival, participants were guided into the simulator control room, where they were briefed on lab safety procedures and filled in the consent form and demographic inventories. Participants were then guided inside the driving simulator. They were informed that the experiment would start by recording their resting physiological state baseline for 4 minutes, and after that, the driving scenario would begin. They were asked to remain seated in the driver’s seat, not move excessively, to breathe normally, and stay relaxed during the baseline recording. Participants were advised that the experimenter would inform them of the start and end of the baseline recording. The driving simulator lights were switched off, the room was silent, and driving scenarios were not projected on the screen. The fNIRS data recording hardware was placed in a backpack behind the driver’s seat (Figure 3). The fNIRS headset was attached when the participant was seated and calibrated to start the baseline recording. After recording the baseline, the automated driving scenarios lasted approximately 5 minutes. The total duration of the experiment was 11 minutes and 30 seconds.

Figure 3.

fNIRS equipment. In our experiment, we attached the backpack to the back of the driver’s seat. Source: nirx.net.

Participants were instructed to sit in the driver’s seat but were not explicitly asked to monitor the environment. Instead, they were asked not to engage in the driving task. The rationale for doing this was that they were about to test a highly automated vehicle that they did not need to drive manually, nor would they be requested to take over. Participants were not free to perform other tasks, as this could disrupt their situation awareness or affect their neurophysiology.

The automated driving scenarios were split into two segments. An initial two-minute suburban driving scenario labelled Driving Condition 1 was split into 30-second blocks for fNIRS analysis, with interblock intervals of 15 seconds (Figure 4). These segments were thus referred to as DC1.1, DC1.2, and DC1.3. Driving scenarios started with the ego vehicle stopped at a red traffic light at a five-lane roundabout, which carries traffic to and from the highway to the suburbs and the city centre. This initial portion of the scenario lasted 60 seconds and served as a familiarisation so participants could adjust to the driving simulation. The ego vehicle took the third roundabout exit leading to a straight dual carriageway, separated by a central reservation. Speed was limited from 30 to 50 mph. Surrounding traffic levels were low (<5 road users per minute), and weather conditions were cloudy. Approximately 1 minute later, the ego vehicle entered the suburbs. This layout consisted of two lanes passing through residential areas at a maximum of 30 mph, including several left and right turns and give-way exits. Oncoming traffic increased to medium levels (<20 road users per minute), including pedestrians, cyclists, and parked cars, on the roadside and driveways.

Figure 4.

Experimental conditions in order of occurrence. Shaded boxes indicate data analysed.

The simulation continued with a two-minute city centre scenario, denoted as Driving Condition 2, that is, DC2.1, DC2.2, and DC2.3, as these were also split into 30-second blocks with interblock intervals of 15 seconds (Figure 3). In this scenario, the layout changed to a ‘high street’ area surrounded by commercial buildings, signs, and billboards. It also implied higher levels of moving pedestrians and vehicles, including vans and buses, stopped on the roadside – which the vehicle had to overtake – and T-junctions with traffic approaching from both directions (between 20 and 40 road users per minute). The speed limit was 30 mph, and the weather shifted to heavy rain, degrading the visual range.

Finally, the Driving Hazard event occurred when leaving the city centre to enter the suburbs again, on the approach of a T-junction, in a residential area from a straight two-way lane. This event was the sudden appearance of a heavy single-cabin semitrailer truck, which accelerated into the scene at high speed (60 mph) from the left side of the T-junction ahead, moving sideways and heading directly towards the ego vehicle. The ego vehicle performed a sudden evasive manoeuvre to avoid the trailer, steered to the right side and collided with a garden fence. This sequence (i.e., from leaving DC2 to the end of the crash) occurred over 30 seconds. After the hazardous event, participants remained in the vehicle with the scenario displayed on-screen for 2 minutes to record a postevent recovery. Afterwards, the experimenter entered the simulator and accompanied them back into the control room to fill in the risk perception and the trust in automation scale.

Data Preprocessing

Raw fNIRS data were preprocessed using HomER 3 (Huppert et al., 2009) scripts running on MATLAB R2019a (Mathworks Inc.) according to the current recommendations for preprocessing fNIRS data (Pinti et al., 2019) (Table 2). For current best practices and publication guidelines see Yücel et al., 2021. Corrected optical density data were then converted to HbO, HbR, and HbT concentrations using the modified Beer–Lambert law. Once optical density concentrations were calculated, data was block-averaged and exported as haemodynamic response function (HRF) means.

TABLE 2.

Data Preprocessing

Step	Description	Function	Input Values
1	Remove channels in which the signal was too weak, too strong, or their standard deviation was too great	hmrRPruneChannels	dRange = 1 × 10⁴ 1e + 07 SNRthresh = 5 SDrange = 0.0 to 45.0
2	Transforms fNIRS raw data into optical density	hmrRIntensity2OD	None
3	Identifies and corrects motion artefacts	hmrRMotionCorrectPCArecurse	tMotion = 0.5 tMask = 1 STDthresh = 20 AMPthresh = 5 nSV = 0.97 maxIter = 5
4	Eliminates noise from physiological activity, low-frequency signal drifts or machine noise.	hmrRBandPassFilt	hpf = 0.01 lpf = 0.5
5	Converts optical density to concentrations	hmrROD2concNew	ppf = 1 (760 nm) 1 (850 nm)
6	Calculates the block average of the given conditions	hmrR_BlockAvg	None

Data Analysis

Block-averaged HbO and HbR values from HomER 3 were exported in excel files containing HRF means for each channel, condition, and participant. The underlying ROIs were determined using the NIRS Brain AnalyzIR toolbox (Santosa et al., 2018) to calculate the corresponding anatomical labels for each position. The toolbox creates a variable that lists the channels and BAs covered by the probe and the relative ‘weights’ for each channel and BA. The weights for each BA add up to 1. The channel with the most sensitivity to a BA has the highest weight for that area. The relative weight is a helpful metric, but it does not give the complete picture, so we also extracted a ‘depth’ value for each channel and BA. Depth values represent the distance on average between the channel and the BA – that is, the further the distance, the lower the likelihood that the channel captures that BA. Therefore, we selected up to three channels accounting for at least a combined relative weight of 0.80 (i.e., covering at least 80% of a particular ROI) and for the lowest combined depth value (i.e., the smallest combined distance on average).

The rationale for not averaging all channels with a relative weight greater than 0 for a given BA is that some values are too low. If too many channels are averaged together, the response will be negated. Following Wiggins et al. (2016), we established averaging only up to 3 channels together. The most sensitive channels of each ROI were grouped. This led to 10 ROIs: Bilateral BAs 08, 09, 10, and 46, and left BA44 and 45. Having grouped the relevant channels into ROIs, values were averaged within each ROI for each experimental condition, resulting in seven means (one per experimental condition) per participant for each ROI and each chromophore (Table 1). These concentration values were then standardised to enable interindividual and intraindividual comparisons using Z-scores (M = 0; SD = 1). Each single mean concentration value was then transformed into Z-scores against the mean group baseline value and its standard deviation (i.e., Z = (X – baseline mean)/baseline SD) (Table 3). Data standardisation is a common procedure among fNIRS studies to allow for interindividual comparisons in parametrical statistical analysis using block-averaged values (Durantin et al., 2014; Leon-Dominguez et al., 2014; Lin et al., 2020; Minematsu et al., 2018; Roche-Labarbe et al., 2008; Tanida et al., 2004; Verdière et al., 2018).

TABLE 3.

Data Analysis

Step	Description	Function	Criteria
1	Determining underlying ROIs for each channel	nirs.util.converlabels2roi
2	Determining sensitivity for each channel using relative weights and depth values	nirs.util.depthmap
3	Determining most sensitive channels combined for each ROI		Up to three channels with the highest relative weight and lowest depth values
4	Averaging most sensitive channels for each ROI and for each experimental condition	Mean
5	Standardising individual mean HRF concentrations	Z = (X – baseline mean)/baseline SD
5	Test for normality assumption	Shapiro–Wilk	p > 0.05
6	Test for sphericity assumption	Mauchly	p > 0.05
7	Test for main statistical effects and interactions using mixed repeated-measures ANOVA	Analysis of variance	p < 0.05
8	Follow-up pair-wise comparisons	Bonferroni correction	p < 0.05

The analysis of variance (ANOVA) is a common technique to determine localised brain activation based on changes in simultaneous HbO and HbR concentrations in repeated measures and block designs (Balters et al., 2021; Tak & Ye, 2014). Although it is common in the literature to report only HbO, HbR or HbT, the haemodynamic response is bi-dimensional. HbO and HbR usually correlate negatively during brain stimulation because increased blood flow produces an increase in oxygenated haemoglobin and a decrease in deoxygenated haemoglobin (Fallgatter & Strik, 1998; Mehagnoul-Schipper et al., 2002; Schroeter et al., 2002; Taga et al., 2003). Nonetheless, since these features may not necessarily be reciprocal, several authors have argued that interpretations based exclusively on one chromophore would be incomplete and advocate reporting both features in tandem (Liu et al., 2016; Obrig, 2010; Quaresima et al., 2012). Therefore, following these recommendations, repeated measures ANOVAs were conducted to determine changes in haemoglobin concentrations on each chromophore separately.

HbO, HbR, and HbT mean HRF concentrations grouped in ROIs were imported and analysed with IBM SPSS Statistics 26 software. The significance level was set at α < 0.05. The Shapiro–Wilk’s test (p ≥ 0.05) was used to assess normality assumption violations, and Mauchly’s test was used to assess the assumption of sphericity. Thus, repeated-measures ANOVAs were conducted for each ROI individually (i.e., BAs 8, 9, 10, and 46 bilateral, plus BAs 44 and 45 on the left hemisphere). Main effects and interactions were followed-up by Bonferroni-corrected pair-wise comparisons.

RESULTS

This block design experiment investigated whether variations in perceived risk from either slowly evolving or rapidly evolving driving conditions would produce observable changes in neurophysiology. Two participants were excluded from the analysis due to significant noise in raw data (N = 18).

Hypothesis 1 – Perceived Risk From Slowly Evolving Conditions

This hypothesis tested whether slow changes across Driving Conditions would produce oxygenation concentration variations within participants. We ran a repeated-measures ANOVA with 8 levels (BL, DC1.1, DC1.2, DC1.3, DC2.1, DC2.2, DC2.3, and Recovery; see Figure 4) on HbO, HbR and HbT, but HbR did not report any statistically significant effects.

A main effect for Driving Conditions was observed in the left orbitofrontal cortex (BA10-L) for HbO (F (7, 119) = 2.330, p = 0.029, η²p = 0.121, Figure 5). Post hoc tests indicated an increase in oxygenation from BL (0.000 ± 1.000) to DC2.1 (1.067 ± 0.880, p = 0.020).

Figure 5.

HbO levels in BA10-left and BA45-left (top), and HbT levels in BA45-left, BA46-left (bottom); across moderate risk driving conditions. Asterisks (*) indicate main effects for condition. Mean is indicated by (x). Error bars indicate standard error.

BA45, located in the left ventrolateral cortex, reported a main effect for Driving Conditions on HbO (F (7, 119) = 2.197, p = 0.039, η²p = 0.114, Figure 5). Post hoc tests indicated an increase in oxygenation from BL (0.000 ± 1.000) to DC2.1 (1.093 ± 0.660, p = 0.011). Furthermore, HbT levels also varied within Driving Conditions (F (7, 119) = 2.827, p = 0.032, η²p = 0.143, Figure 5), with post hoc comparisons revealing an increase from BL (0.000 ± 1.000) to DC1.2 (0.829 ± 0.671, p = 0.039) and DC2.1 (1.061 ± 0.554, p = 0.008).

BA46-L, located in the left dorsolateral cortex, reported a main effect for Driving Conditions on HbT (F (7, 119) = 2.902, p = 0.037, η²p = 0.146, Figure 5), with post hoc comparisons revealing an increase from DC1.3 (0.372 ± 0.592) to DC2.1 (1.006 ± 0.462, p = 0.049).

A further exploration with participant sex as independent variable reported a main effect on the right anterior premotor cortex (BA08-R) for HbT (F (1, 16) = 4.541, p = 0.049, η²p = 0.221). Pair-wise comparisons revealed that Women registered significantly higher levels of HbT (0.199 ± 0.407, p = 0.011) than Men (−1.390 ± 1.609) during the postevent recovery period Figure 6.

Figure 6.

HbT levels in BA08 – right between men and women across moderate traffic complexity driving conditions. Double asterisks (**) indicate main effects between subjects. Mean is indicated by (x). Error bars indicate standard error.

Hypothesis 2 – Perceived Risk From Rapidly Evolving Conditions

The second hypothesis investigated whether the rapidly evolving Driving Hazard event would produce observable effects in oxygenation concentrations within participants. To analyse the effect of the rapidly evolving Driving Hazard event (H2), we ran a repeated measures ANOVA with three levels (BL, Driving Hazard, and Recovery).

Strong evidence in favour of H2 was found across HbO and HbT; however, HbR did not report any statistically significant effect. A main effect for Driving Hazard on HbO (F (2, 34) = 4.418, p = 0.020, η²p = 0.206) and HbT (F (2, 34) = 3.470, p = 0.043, η²p = 0.170) was found on BA09-R, but these effects diminished in post hoc tests.

Lateralised orbitofrontal activation was observed, with BA10-R reporting a main effect of Driving Hazard on HbO (F (2, 34) = 5.846, p = 0.007, η²p = 0.256, Figure 7) and post hoc tests revealing an increase from BL (0.000 ± 1.000) to Hazard (1.451 ± 1.661, p = 0.021). HbT also showed this effect (F (2, 34) = 4.118, p = 0.025, η²p = 0.195), although diminishing with post hoc tests.

Figure 7.

HbO levels in BA10-right and BA45-left (top), and in BA44-left, BA46-left (bottom); between the driving hazard condition and resting periods. Asterisks (*) indicate main effects for condition. Mean is indicated by (x). Error bars indicate standard error.

BA44 reported a main effect of Driving Hazard on HbO (F (2, 34) = 6.968, p = 0.007, η²p = 0.291, Figure 7) with post hoc tests revealing an increase from BL (0.000 ± 1.000) to Hazard (1.145 ± 1.530, p = 0.019). HbT seconded this effect (F (2, 34) = 6.691, p = 0.011, η²p = 0.282, Figure 8), with post hoc tests revealing a similar increase from BL (0.000 ± 1.000) to Hazard (1.111 ± 1.579, p = 0.030).

Figure 8.

HbT levels in BA44-left, BA45-left, and BA46-left between the driving hazard condition and resting periods. Asterisks (*) indicate main effects for condition. Mean is indicated by (x). Error bars indicate standard error.

BA45 reported a main effect of Driving Hazard on HbO (F (2, 34) = 10.950, p < 0.001, η²p = 0.392, Figure 7) with post hoc tests revealing an increase from BL (0.000 ± 1.000) to Hazard (1.387 ± 1.455, p = 0.002), and followed by a decrease from Hazard to Recovery (0.436 ± 1.060, p = 0.038). HbT seconded this effect (F (2, 34) = 7.559, p = 0.002, η²p = 0.308, Figure 8), with post hoc tests revealing a similar increase from BL (0.000 ± 1.000) to Hazard (1.162 ± 1.503, p = 0.013).

The dorsolateral prefrontal cortex showed bilateral activity during the Driving Hazard event. BA46-L reported a main effect of Driving Hazard on HbO (F (2, 34) = 11.743, p < 0.001, η²p = 0.409, Figure 7) with post hoc tests revealing an increase from BL (0.000 ± 1.000) to Hazard (1.434 ± 1.227, p = 0.002), and followed by a decrease from Hazard to Recovery (0.427 ± 0.916, p = 0.017). HbT seconded this effect (F (2, 34) = 9.008, p = 0.001, η²p = 0.346, Figure 8), with post hoc tests revealing a similar increase from BL (0.000 ± 1.000) to Hazard (1.356 ± 1.386, p = 0.009), and a posterior decrease from Hazard to Recovery (0.467 ± 0.898, p = 0.046). BA46-R reported a main effect of Driving Hazard on HbO (F (2, 34) = 5.760, p = 0.017, η²p = 0.253), although fading away with post hoc comparisons.

Finally, a further exploration with sex as independent variable reported a main effect for HbT in the right dorsolateral prefrontal cortex (BA46-R) (F (1, 16) = 5.590, p = 0.031, η²p = 0.259 Figure 9); however, this effect diminished after pair-wise comparisons. Nonetheless, it is worth noting that descriptive data indicate higher levels of HbT on women (1.142 ± 1.088) than men (0.380 ± 1.274) during the event.

Figure 9.

HbT levels in BA46 - right between men and women during the driving hazard condition and resting periods. Mean is indicated by (x). Error bars indicate standard error.

Psychometric Results

Finally, a Wilcoxon signed-rank test reported a main effect for self-reported risk perception (Z = 194.5, p = 0.001), with perceived risk during the Driving Hazard event (Mdn = 5.50, IQR = 3) being significantly higher than during Driving Conditions (Mdn = 3.00, IQR = 3).

Results from the TASS scale revealed a main effect for trust (F (1, 18) = 5.975, p = 0.025, η²p = 0.249), indicating a significant increase in trust ratings after the trial (4.721 ± 1.207, p = 0.025) compared to before the trial (4.279 ± 0.893).

DISCUSSION

Hypothesis 1

This empirical research investigates the effect of traffic complexity on drivers’ perceived risk measured during highly automated driving through prefrontal haemoglobin oxygenation concentrations.

The first hypothesis investigated whether suburban and urban driving conditions with moderate levels of traffic complexity and potential unmaterialised hazards would produce variations in brain oxygenation within participants. Moderate risk during these was assumed as the median of self-reported perceived risk across Driving Conditions was 3 out of a maximum of 7.

The null hypothesis was rejected since oxygenated and total haemoglobin (HbO and HbT) concentrations increased significantly in the orbitofrontal (BA10), ventrolateral (BA45) and left dorsolateral (BA46) areas would be consistent with supervising the automated vehicle’s performance when transitioning from the suburbs (DC1.2 and DC1.3) to the city centre (DC2.1) (Figure 5).

The orbitofrontal region has been found to activate when operators are judging the trustworthiness of automated systems (Palmer et al., 2020; Perello-March et al., 2023) and as an indicator of the willingness to intentional engagement (Dimoka, 2010), thus perhaps indicating an intention of taking-over manual control during these scenarios.

Increased oxygenation in the ventrolateral cortex has been associated with distrust (Hubert et al., 2018; Palmer et al., 2020), state-level suspicion (Hirshfield et al., 2014), and frustration during automated driving (Damm et al., 2019). In addition, this area is related to intense negative emotions of distrust, fear and negative consequences (Dimoka, 2010; Hubert et al., 2018).

The left dorsolateral region reported HbT variations from the suburbs (DC1.3) to the city centre (DC2.1). This area is implicated in decision-making from perceptual inputs and has been associated with trust calibration (Drnec & Metcalfe, 2016; Hubert et al., 2018; Palmer et al., 2020). Perhaps participants perceived the change in traffic complexity between driving conditions 1 and 2 (i.e., from suburbs to the city centre) and evaluated the automated vehicle performance and trustworthiness when entering such a new driving context. However, self-reported data showed that perceived risk across driving conditions remained moderate.

Finally, an exploration of sex differences indicated that women registered significantly higher levels of HbT in the right anterior premotor cortex (BA08) during the recovery phase. This area has been found to activate when participants experience uncertainty (Volz et al., 2005). Moreover, these results would align with previous work finding that females show increased lateralised right cortical activity and are more alert during complex traffic conditions (Foy et al., 2016).

Findings from H1 suggest drivers actively monitored the road for potential hazards during moderate traffic complexity conditions. Drivers showed increased dorsolateral, ventrolateral, and orbitofrontal cortical oxygenation that could be attributed to the awareness of un-materialised potential road hazards. Considering that they were not actively instructed to remain responsible for the driving task, we suggest the increased activation in these cortical areas could be attributed to suspicion towards the vehicle’s trustworthiness and the evaluation of the likelihood of a crash – that is, the first component of risk perception – and therefore, participants could be actively calibrating their trust in the automated vehicle.

Hypothesis 2

The second hypothesis predicted that the materialised hazard in a quickly escalating driving scenario requiring an evasive manoeuvre would significantly increase haemoglobin oxygenation compared to baseline resting and postevent recovery resting. Substantial evidence favouring hypothesis 2 was found as participants reported significantly greater risk during the Driving Hazard (Mdn = 5.5/7) than during the Driving Conditions (Mdn = 3/7), supported by robust increases throughout Hbo and HbT.

As in hypothesis 1, bilateral orbitofrontal (i.e., BA10) activation during the Driving Hazard could be attributed to actively judging the automated vehicle trustworthiness derived from an increased risk perception (Dimoka, 2010; Palmer et al., 2020). Consistent activation of this cortical area due to factors related to automated driving performance – for example, traffic complexity and driving conditions – also in Perello-March et al. (2023) reinforce the notion that the orbitofrontal cortex plays a crucial role in judging the trustworthiness of automated vehicles during uncertain situations – that is, trust calibration. This cortical region likely acts as a ‘comparator’ for perceptual information and vehicle reliability, from which situational trust is derived.

Further evidence supporting this statement was found in the left ventrolateral prefrontal cortex (BA44/BA45). Increments in both HbO and HbT from baseline to Hazard scenarios are indicative of attention allocation during visual search (Anderson et al., 2007), increment in distrust (Hirshfield et al., 2014; Hubert et al., 2018; Palmer et al., 2020), and experiencing strong unpleasant emotions (Hirshfield et al., 2014; Hoshi et al., 2011; Hubert et al., 2018). Likely, our participants actively sought visual cues to analyse, understand and predict the potential consequences of the sudden Driving Hazard event, which might be the anticipation of unpleasant emotions such as momentary fear and distrust.

Notably, the insular cortex in the inferior frontal gyrus – near BA45 – is often called the centre for risk perception. It has been associated with decisions with strong negative emotional components (Dimoka, 2010; Hubert et al., 2018), possibly an evolutionary trace to prevent negative interactions and their consequences (Kahneman & Tversky, 1983, 2019; Rangel et al., 2008). Moreover, the insula seems strongly related to a cognition-based mechanism for risk assessment of contextual information and their appraisal (Hubert et al., 2018; Singer et al., 2009). BA44/45 are close to the insular cortex.

Bilateral activation was also observed in the dorsolateral prefrontal cortex (i.e., BA46-L, BA46-R, and BA09-R), reporting HbO and HbT increases from baseline to Hazard, followed by decreases from Hazard to Recovery phases in both measures. Dorsolateral prefrontal cortex activity is often attributed to deliberate decision-making and reflective processes related to trust (Dimoka, 2010; Drnec et al., 2016; Hubert et al., 2018) and supporting the orbitofrontal cortex in comparing uncertain perceptual – for example, visuospatial – information (Bruno et al., 2018). Therefore, as in the orbitofrontal cortex, higher activation of the dorsolateral prefrontal cortex would be related to assessing the trustworthiness of the automated vehicle when relevant contextual changes occur (Hubert et al., 2018; Perello-March et al., 2023). Besides, the dorsolateral prefrontal cortex has been attributed to play an essential role in emotional regulation and self-control (Hirshfield et al., 2014; Hubert et al., 2018). Activation of this area after experiencing a sudden strong negative emotion – fright, startle, fear – derived from the Hazard event would indicate emotional regulation. Supporting this claim, we observed a significant decrease in HbO and HbT during the recovery phase.

Finally, the evaluation of sex-based differences in cortical oxygenation showed increased HbT levels in the right dorsolateral prefrontal cortex (BA46) in women during the recovery phase. This finding aligns with those in H1, indicating a lateralised right dorsolateral and premotor cortex activation in women not present in men. As mentioned, prefrontal hemispheric lateralisation for women during hazardous driving conditions was also observed in an fNIRS study conducted by Foy et al. (2016).

Results from H2 indicate that drivers perceived the hazard but also evaluated the likelihood and the severity of the outcome of the crash. This was inferred since drivers reported higher risk and we observed the activation of areas related to emotion regulation and the anticipation of negative consequences derived from the severity of the crash. Since drivers were freed from the real-time driving task, they could allocate cognitive resources to evaluate how well the driving automation could handle the situation and apply an appropriate action. Hence, we suggest that contrary to manual driving, when drivers evaluate the ability of the highly automated driving to handle a materialised hazardous situation, both components of risk perception – the likelihood and the severity of the crash – are present.

These results build on previous neurophysiology and traffic psychology work using fNIRS relating risk-taking behaviours while driving and age, with young males showing a lack of prefrontal maturation that may explain the increased crash risk seen in this population (Foy et al., 2016). Measuring drivers’ prefrontal activation with automated driving could be used to investigate risk-taking behaviours in future work. Whereas risky driving behaviours have been extensively investigated in previous traffic behaviour psychology (Jonah, 1997), little work has investigated risk-taking behaviours and driver hazard perception under automated driving. Therefore, our findings are of particular novelty and relevance to shed light on this technology’s new human factors–related challenges and comprehend how automated driving users perceive risks.

In addition, we observed an increase in the self-reported trust after the trial. Whilst this finding may seem contradictory to the fact that perceived risk was higher towards the end of the trial. One could expect trust to be low in such a context. It can be argued that the TASS scale measures propensity to trust (i.e., dispositional trust) rather than situational trust (Adams et al., 2003; Holthausen et al., 2020). Hence, such increment in dispositional-learned trust could be attributed to familiarisation due to mere exposure to the automated driving system and the fact that the hazardous event was negotiated successfully. Similar trust increments after a short exposure to automated driving have also been reported in previous studies (Dixon et al., 2019; Gold et al., 2015; Kraus et al., 2020; Kundinger et al., 2019; Large et al., 2019; Lee et al., 2021). The findings evidenced in this paper suggest that situational perceived risk does not necessarily affect dispositional and learned trust, which other factors would modulate (for a review of trust layers see Hoff & Bashir, 2015). Whereas a higher situational risk perception is expected to correlate negatively with lower situational trust (Li et al., 2019), this may not necessarily apply to dispositional and learned trust.

Our results demonstrate that fNIRS is valid for measuring variations in perceived risk from traffic complexity during highly automated driving, particularly HbO and HbT measures. The lack of significant effects for HbR data is not unusual since HbR is known to be a less robust parameter (Balters et al., 2021). Increased oxygenation in prefrontal areas would indicate active monitoring of the driving performance reliability in complex traffic scenarios. Even though our drivers were not responsible for the driving task and dispositional trust was high, they showed variations in perceived risk as traffic complexity built up gradually, especially during the driving hazard. This suggests our drivers were ‘in the loop’ and would have been able to resume manual control if required.

Limitations and Future Work

Using a bespoke questionnaire to assess perceived risk may have limited our qualitative data to the second component of risk perception – that is, the severity of the outcome – which is present when drivers assess perceived with hindsight (Borowsky & Oron-Gilad, 2013). Given that the first component – that is, the likelihood of the crash – is mainly available in real-time, perhaps future work should consider implementing a button press response when a hazard has been identified by the driver. In addition, not counterbalancing the experimental conditions may have induced order effects. However, we deemed it necessary to ensure the driving was immersive and realistic. In addition, recalibrating the fNIRS signal for each block was not feasible because it would have extended the length of the simulation and potentially fatigued participants or induced motion sickness.

Although brain activity measures such as fNIRS are not likely to be integrated into production driver monitoring systems (DMS) in the short term – at least not with current wearables – this paper has proven fNIRS to be a helpful research tool for assessing drivers' states. In terms of practical advantages, compared to other common noninvasive measures such as eye-tracking or peripheral physiology, fNIRS can provide a direct measure of complex driver states such as situation awareness (Bracken et al., 2021), trust (Perello-March et al., 2023), driver attention allocation to take-over requests (Fu et al., 2020), or out-of-the-loop states (Balters et al., 2017). fNIRS offers a nearly real-time measure – that is, the haemodynamic response takes only 3–5 seconds – of cortical responses to specific events that can be mapped on the brain regions responsible for certain tasks. For example, fNIRS can be used to measure drivers’ workload during a manual take-over control by mapping prefrontal areas (i.e., known to be responsible for decision-making, working memory, anticipation or judging scenarios) in combination with the motor cortex (responsible for manual tasks or coordinating haptic responses), or measure drivers’ situation awareness levels to visual cues on the road by measuring the activity on the visual cortex (responsible for visual object detection and interpretation). Hence, we encourage future work in this domain to use fNIRS as a complement to gaze-behaviour indicators, behavioural measures, and self-reports or to other physiological indicators to understand better the new challenges arising from automated driving technology.

CONCLUSION

Overall, this empirical research has evidenced traffic complexity affects risk perception and its derived neurophysiological indicators. Brain oxygenation measures successfully evaluated moderate and high perceived risk levels, indicating drivers actively supervised the vehicle operation in complex traffic scenarios. Our findings evidence the benefits of using fNIRS for driving research to assess driver states and risk in real-time.

KEY POINTS

Traffic complexity affects the perception of risk in highly automated driving.

Increased perception of risks leads to an increase in active monitoring and supervision behaviours with a HAV.

HAD frees up cognitive resources to evaluate both the likelihood and severity of a possible crash event.

Females remained more vigilant than males after the driving hazard event.

fNIRS has proven to be a valuable tool for driver state monitoring.

Footnotes

Acknowledgements

This work was supported by the University of Warwick (PhD studentship).

ORCID iDs

Jaume Perello-March

Roger Woodman

Mark T. Elliott

Jaume R. Perello-March received the B.Sc. degree in psychology and the MSc degree in human evolution and cognition from the University of the Balearic Islands, Spain, in 2014 and 2016, respectively, and the PhD degree in Human Factors engineering from the University of Warwick in 2022. He holds previous automotive research and industrial experience in leading European centres, such as the Swedish National Road Transport Research Institute (VTI) and the Galician Automotive Technology Centre (CTAG). He is currently a Research Fellow in human factors for future transport with the National Transport Design Centre (NTDC), Coventry University. He has articles published in his field in Q1/4 journals and has presented his work at international conferences. His research interests include driver state monitoring and behaviour.

Christopher G. Burns received the BSc, Master by Research (M.Res.), and PhD degrees in psychology from The University of Edinburgh, with a specialist background in quantitative research methods and statistics in individual differences with psychometric and psychophysiological methods. He joined the WMG’s Human Factors Section, as a Postdoctoral Research Fellow on the UK Autodrive Project, embedded with a Research Team at Jaguar Land Rover, in 2017. His involvement in UK Autodrive, consisted of designing and refining methodologies, then conducting and analysing practical experiments in passenger and pedestrian experiences of a prototype autonomous low-speed electric vehicle (L-SATS) operating in a controlled arena. These studies included user attitudes and intentions-to-use, trust formation, technology acceptance and internal/external human–machine interface evaluations, using both quantitative and qualitative methods. Previously, he worked on projects involving driving simulation, sustained attention, mental workload, and emotional and attitudinal responses. He previously worked on ultrasonographic simulation and medical training in teaching contexts.

Roger Woodman received the PhD degree from the Bristol Robotics Laboratory in 2013. He is currently an Assistant Professor and the Human Factors Research Lead at WMG. He is also the Deputy Director of the Centre for Doctoral Training, WMG, training doctoral researchers in the areas of intelligent and electrified mobility systems. He lectures on the topics of human factors of future mobility. He has more than 20 years of experience working in industry and academia. He has several scientific articles published in the field of connected and autonomous vehicles. His research interests include self-driving vehicles, shared mobility, and human–machine interaction.

Stewart A. Birrell received the BSc degree in sport science and the PhD degree in ergonomics from Loughborough University, U.K., in 2002 and 2007, respectively. He is currently a Professor of human factors for future transport with the National Transport Design Centre (NTDC), Coventry University. He has spent the previous 15 years working within the transportation sector within industry and academia, with expertise ranging from driver behaviour and distraction, multimodal warnings, user state monitoring and information requirements—all underpinned by the design of in-vehicle information systems, and their evaluation using driving simulators, virtual reality (VR), and field operational trials. He also applies innovative human factors engineering methodologies to enable real world and virtual evaluation of user interaction with connected and autonomous vehicle (CAV), electric vehicle (EV), and urban air mobility (UAM) technologies and services. He has over 100 journals and conference papers, book sections and articles published in his field to date. He is an Editor of the internationally renowned, Q1/4∗ journal IEEE TRANSACTIONS ON INTELLIGENT TRANSPORTATION SYSTEMS.

Mark T. Elliott received the M.Eng. degree in electronic systems engineering and the Ph.D. degree in biomedical engineering from Aston University in 2000 and 2007, respectively, developing intelligent systems to discriminate between different walking patterns. He is currently an Associate Professor at the Institute of Digital Healthcare, WMG, University of Warwick. He subsequently spent a number of years as a Research Fellow with the Sensory Motor Neuroscience Laboratory, University of Birmingham, modelling multisensory integration in the context of human movement coordination. His research interests include measuring health, wellbeing, and behaviour through data- driven approaches. This primarily involves analysing and modelling data from wearable and mobile devices that capture movement and physiological responses. Much of his research is highly applied and involves collaborating with commercial and public sector partners.

References

Adams

B. D.

Bruyn

L. E.

Houde

Angelopoulos

(2003). Trust in automated systems: Literature review (Issue CR-2003-09).

Alrefaie

M. T.

Summerskill

Jackon

T. W.

(2019). In a heart beat: Using driver’s physiological changes to determine the quality of a takeover in highly automated vehicles. Accident Analysis and Prevention, 131, 180–190. https://doi.org/10.1016/j.aap.2019.06.011

Anderson

E. J.

Mannan

S. K.

Husain

Rees

Sumner

Mort

D. J.

McRobbie

Kennard

(2007). Involvement of prefrontal cortex in visual search. Experimental Brain Research, 180(2), 289–302. https://doi.org/10.1007/s00221-007-0860-0

Balters

Baker

J. M.

Geeseman

J. W.

Reiss

A. L.

(2021). A Methodological Review of fNIRS in Driving Research: Relevance to the Future of Autonomous Vehicles. In Frontiers in Human Neuroscience (Vol. 15, p. 94). Frontiers Media S.A. https://doi.org/10.3389/fnhum.2021.637589

Balters

Sibi

Johns

Steinert

(2017). Learning-by-doing: Using near infrared spectroscopy to detect habituation and adaptation in automated driving. AutomotiveUI 2017 - 9th International ACM Conference on Automotive User Interfaces and Interactive Vehicular Applications, Proceedings, 17, 134–143. https://doi.org/10.1145/3122986.3123006

Borowsky

Oron-Gilad

(2013). Exploring the effects of driving experience on hazard awareness and risk perception via real-time hazard identification, hazard classification, and rating tasks. Accident Analysis and Prevention, 59, 548–565. https://doi.org/10.1016/j.aap.2013.07.008

Bracken

Tobyne

Winder

Shamsi

Endsley

M. R.

(2021). Can situation awareness be measured physiologically? In H

(Eds.), Adva1gnitive engineering. AHFE 2021. Lecture notes in networks and systems. (Vol. 259, pp. 31–38). Springer. https://doi.org/10.1007/978-3-030-80285-1_4

Bruno

J. L.

Baker

J. M.

Gundran

Harbott

L. K.

Stuart

Piccirilli

A. M.

Hosseini

S. M. H.

Gerdes

J. C.

Reiss

A. L.

(2018). Mind over motor mapping: Driver response to changing vehicle dynamics. Human Brain Mapping, 39(10), 3915–3927. https://doi.org/10.1002/hbm.24220

Damm

Fränzle

Lüdtke

Rieger

J. W.

Trende

Unni

(2019). Integrating neurophysiological sensors and driver models for safe and performant automated vehicle control in mixed traffic∗. In IEEE Intelligent Vehicles Symposium, Proceedings, 2019-June, 82–89. https://doi.org/10.1109/IVS.2019.8814188

10.

Dimoka

(2010). What does the brain tell us about trust and distrust? Evidence from a functional neuroimaging study. MIS Quarterly, 34(2), 373–396. http://www.jstor.org/stable/20721433

11.

Dixon

Megill

W. M.

Nebe

(2019). Trust in automation: An on-road study of trust in advanced driver assistance systems. In VEHICULAR 2019: The Eighth International Conference on Advances in Vehicular Systems, Technologies and Applications, c (pp. 85–93). https://www.thinkmind.org/index.php?view=article&articleid=vehicular_2019_6_10_30041

12.

Dong

Uchimura

Murayama

(2011). Driver inattention monitoring system for intelligent vehicles: A review. IEEE Transactions on Intelligent Transportation Systems, 12(2), 596–614. https://doi.org/10.1109/TITS.2010.2092770

13.

Drnec

Marathe

A. R.

Lukos

J. R.

Metcalfe

J. S.

(2016). From trust in automation to decision neuroscience: applying cognitive neuroscience methods to understand and improve interaction decisions involved in human automation interaction. Frontiers in Human Neuroscience, 10(June), 1–14. https://doi.org/10.3389/fnhum.2016.00290

14.

Drnec

Metcalfe

J. S.

(2016). Paradigm development for identifying and validating indicators of trust in automation in the operational environment of human automation integration. In Schmorrow

Fidopiastis

(Eds.), Lecture notes in computer science (foundations of augmented cognition) (Vol. 9744, pp. 157–167). Springer International Publishing AG. https://doi.org/10.1007/978-3-642-02812-0

15.

Chang

(2020). The interactive effect of vehicle signals and sensation-seeking on driver hazard perception. Transportation Research Part F: Traffic Psychology and Behaviour, 73, 174–187. https://doi.org/10.1016/j.trf.2020.06.018

16.

Durantin

Gagnon

J. F.

Tremblay

Dehais

(2014). Using near infrared spectroscopy and heart rate variability to detect mental overload. Behavioural Brain Research, 259, 16–23. https://doi.org/10.1016/j.bbr.2013.10.042

17.

Endsley

M. R.

(1995). Toward a theory of situation awareness in dynamic systems. Human Factors, 37(1), 32–64. https://doi.org/10.1518/001872095779049543

18.

Endsley

M. R.

(2017). From here to autonomy: Lessons learned from human-automation research. Human Factors, 59(1), 5–27. https://doi.org/10.1177/0018720816681350

19.

EuropeanCouncil . (2019). Safer cars in the EU. https://www.consilium.europa.eu/en/press/press-releases/2019/11/08/safer-cars-in-the-eu/

20.

Fallgatter

A. J.

Strik

W. K.

(1998). Frontal brain activation during the Wisconsin Card Sorting Test assessed with two-channel near-infrared spectroscopy. Eur Arch Psychiatry Clin Neurosci, 248, 245–249.

21.

Foy

H. J.

Chapman

(2018). Mental workload is reflected in driver behaviour, physiology, eye movements and prefrontal cortex activation. Applied Ergonomics, 73, 90–99. https://doi.org/10.1016/j.apergo.2018.06.006

22.

Foy

H. J.

Runham

Chapman

(2016). Prefrontal cortex activation and young driver behaviour: A fNIRS study. PLoS ONE, 11(5), 1–19. https://doi.org/10.1371/journal.pone.0156512

23.

Hyde

Sibi

Johns

Fischer

Sirkin

(2020). Assessing the effects of failure alerts on transitions of control from autonomous driving systems. In IEEE Intelligent Vehicles Symposium, Proceedings (pp. 1956–1963). https://doi.org/10.1109/IV47402.2020.9304627

24.

Gold

Körber

Hohenberger

Lechner

Bengler

(2015). Trust in Automation – Before and After the Experience of Take-over Scenarios in a Highly Automated Vehicle. Procedia Manufacturing, 3, 3025–3032. https://doi.org/10.1016/j.promfg.2015.07.847

25.

Healey

J. A.

Picard

R. W.

(2005). Detecting Stress During Real-World Driving Tasks Using Physiological Sensors. IEEE Transactions on Intelligent Transportation Systems, 6(2), 156–166. https://doi.org/10.1109/TITS.2005.848368

26.

Hirshfield

Bobko

Barelka

Hirshfield

S. H.

Farrington

M. T.

Gulbronson

Paverman

(2014). Using noninvasive brain measurement to explore the psychological effects of computer malfunctions on users during human-computer interactions. Advances in Human-Computer Interaction, 2014, 1–13. https://doi.org/10.1155/2014/101038

27.

Hoff

K. A.

Bashir

(2015). Trust in automation: Integrating empirical evidence on factors that influence trust. Human Factors, 57(3), 407–434. https://doi.org/10.1177/0018720814547570

28.

Holthausen

B. E.

Wintersberger

Walker

B. N.

Riener

(2020). Situational Trust Scale for Automated Driving (STS-AD): Development and Initial Validation. In Proceedings - 12th International ACM Conference on Automotive User Interfaces and Interactive Vehicular Applications, AutomotiveUI 2020 (pp. 40–47). https://doi.org/10.1145/3409120.3410637

29.

Horswill

M. S.

McKenna

F. P.

(2004). Drivers’ hazard perception ability: Situation awareness on the road. In Banbury

Tremblay

(Eds.), A cognitive approach to situation awareness (pp. 155–175). Ashgate Publishing Ltd. http://centaur.reading.ac.uk/14289/

30.

Hoshi

Huang

Kohri

Iguchi

Naya

Okamoto

Ono

(2011). Recognition of human emotions from cerebral blood flow changes in the frontal region: A study with event-related near-infrared spectroscopy. Journal of Neuroimaging, 21(2), 94–101. https://doi.org/10.1111/j.1552-6569.2009.00454.x

31.

Hubert

Linzmajer

Riedl

Kenning

(2018). Trust me if you can – neurophysiological insights on the influence of consumer impulsiveness on trustworthiness evaluations in online settings. European Journal of Marketing, EJM-12-2016-0870. https://doi.org/10.1108/EJM-12-2016-0870

32.

Huppert

T. J.

Diamond

S. G.

Franceschini

M. A.

Boas

D. A.

(2009). HomER: A review of time-series analysis methods for near-infrared spectroscopy of the brain. Applied Optics, 48(10), D280. https://doi.org/10.1364/AO.48.00D280

33.

Jian

J.-Y.

Bisantz

A. M.

Drury

(2000). Foundations for an empirically determined scale of trust in automated systems. International Journal of Cognitive Ergonomics, 4(1), 53–71. https://doi.org/10.1207/S15327566IJCE0401_04

34.

Joanne

Wetton

M. A.

Horswill

M. S.

Hatherly

Wood

J. M.

Pachana

N. A.

Anstey

K. J.

(2010). The development and validation of two complementary measures of drivers ’ hazard perception ability. Accident Analysis and Prevention, 42(4), 1232–1239. https://doi.org/10.1016/j.aap.2010.01.017

35.

Jonah

B. A.

(1997). Sensation seeking and risky driving: A review and synthesis of the literature. Accident Analysis and Prevention, 29(5), 651–665. https://doi.org/10.1016/S0001-4575(97)00017-1

36.

Kahneman

Tversky

(1983). Choices, Values, and Frames. In MacLean

L. C.

Ziemba

W. T.

(Eds.), Handbook of the fundamentals of financial decision making (Vol. 4, pp. 269–278). World Scientific Publishing. https://doi.org/10.1142/9789814417358_0016

37.

Kahneman

Tversky

(2019). Prospect theory: An analysis of decision under risk. In Choices, values, and frames (Vol. 47, Issue 2, pp. 17–43). https://doi.org/10.1017/CBO9780511803475.003

38.

Kraus

Scholz

Stiegemeier

Baumann

(2020). The more you know: Trust Dynamics and calibration in highly automated driving and the effects of take-overs, system malfunction, and system transparency. Human Factors, 62(5), 718–736. https://doi.org/10.1177/0018720819853686

39.

Kundinger

Riener

Wintersberger

(2019, May 2). (Over)trust in automated driving: The sleeping pill of tomorrow? In Conference on Human Factors in Computing Systems - Proceedings. https://doi.org/10.1145/3290607.3312869

40.

Large

D. R.

Burnett

Salanitri

Lawson

Box

(2019). A longitudinal simulator study to explore drivers’ behaviour in level 3 automated vehicles. In Proceedings - 11th International ACM Conference on Automotive User Interfaces and Interactive Vehicular Applications, AutomotiveUI 2019 (pp. 222–232). https://doi.org/10.1145/3342197.3344519

41.

Lee

Abe

Sato

Itoh

(2021). Developing human-machine trust: Impacts of prior instruction and automation failure on driver trust in partially automated vehicles. Transportation Research Part F: Traffic Psychology and Behaviour, 81, 384–395. https://doi.org/10.1016/j.trf.2021.06.013

42.

Lee

J. D.

Moray

(1992). Trust, control strategies and allocation of function in human-machine systems. Ergonomics, 35(10), 1243–1270. https://doi.org/10.1080/00140139208967392

43.

Leon-Dominguez

Izzetoglu

Leon-Carrion

Solís-Marcos

Garcia-Torrado

F. J.

Forastero-Rodríguez

Mellado-Miras

Villegas-Duque

Lopez-Romero

J. L.

Onaral

Izzetoglu

(2014). Molecular concentration of deoxyHb in human prefrontal cortex predicts the emergence and suppression of consciousness. NeuroImage, 85, 616–625. https://doi.org/10.1016/j.neuroimage.2013.07.023

44.

Holthausen

B. E.

Stuck

R. E.

Walker

B. N.

(2019). No risk no trust: Investigating perceived risk in highly automated driving. In AutomotiveUI ’19: Proceedings of the 11th International Conference on Automotive User Interfaces and Interactive Vehicular Applications, 21–25 September (pp. 177–185). https://doi.org/10.1145/3342197.3344525

45.

Zhang

Dai

Wang

(2009). Assessment of cerebral oxygenation during prolonged simulated driving using near infrared spectroscopy: Its implications for fatigue development. European Journal of Applied Physiology, 107(3), 281–287. https://doi.org/10.1007/s00421-009-1122-6

46.

Lin

C. T.

King

J. T.

Chuang

C. H.

Ding

Chuang

W. Y.

Liao

L. D.

Wang

Y. K.

(2020). Exploring the brain responses to driving fatigue through simultaneous EEG and fNIRS measurements. International Journal of Neural Systems, 30(1), 1–12. https://doi.org/10.1142/S0129065719500187

47.

Liu

Pelowski

Pang

Zhou

Cai

(2016). Near-infrared spectroscopy as a tool for driving research. Ergonomics, 59(3), 368–379. https://doi.org/10.1080/00140139.2015.1076057

48.

Lohani

Payne

B. R.

Strayer

D. L.

(2019). A review of psychophysiological measures to assess cognitive states in real-world driving. Frontiers in Human Neuroscience, 13(57), 1–27. https://doi.org/10.3389/fnhum.2019.00057

49.

Malone

Brünken

(2016). The role of ecological validity in hazard perception assessment. Transportation Research Part F: Traffic Psychology and Behaviour, 40, 91–103. https://doi.org/10.1016/j.trf.2016.04.008

50.

Mehagnoul-Schipper

D. J.

Van Der Kallen

B. F. W.

Colier

W. N. J. M.

Van Der Sluijs

M. C.

Van Erning

L. J. T. O.

Thijssen

H. O. M.

Oeseburg

Hoefnagels

W. H. L.

Jansen

R. W. M. M.

(2002). Simultaneous measurements of cerebral oxygenation changes during brain activation by near-infrared spectroscopy and functional magnetic resonance imaging in healthy young and elderly subjects. Human Brain Mapping, 16(1), 14–23. https://doi.org/10.1002/hbm.10026

51.

Melnicuk

Thompson

Jennings

Birrell

(2021). Effect of cognitive load on drivers’ State and task performance during automated driving: Introducing a novel method for determining stabilisation time following take-over of control. Accident Analysis and Prevention, 151, 1–14. https://doi.org/10.1016/j.aap.2020.105967

52.

Merat

Seppelt

Louw

Engström

Lee

J. D.

Johansson

Green

C. A.

Katazaki

Monk

Itoh

McGehee

Sunda

Unoura

Victor

Schieben

Keinath

(2019). The “Out-of-the-Loop” concept in automated driving: proposed definition, measures and implications. Cognition, Technology and Work, 21(1), 87–98. https://doi.org/10.1007/s10111-018-0525-8

53.

Minematsu

Ueji

Yamamoto

(2018). Activity of frontal pole cortex reflecting hedonic tone of food and drink: fNIRS study in humans. Scientific Reports, 8(1), 16197. https://doi.org/10.1038/s41598-018-34690-3

54.

Muir

B. M.

(1994). Trust in automation: Part I. Theoretical issues in the study of trust and human intervention in automated systems. Ergonomics, 37(11), 1905–1922. https://doi.org/10.1080/00140139408964957

55.

NHTSA . (2022). NHTSA proposes significant updates to five-star safety ratings program. https://www.nhtsa.gov/press-releases/five-star-safety-ratings-program-updates-proposed

56.

Obrig

(2010). From acoustic segmentation to language processing: Evidence from optical imaging. Frontiers in Neuroenergetics, 2(13), 1–12. https://doi.org/10.3389/fnene.2010.00013

57.

Palmer

Richards

Shelton-Rayner

Izzetoglu

Inch

(2020). Assessing variable levels of delegated control – A novel measure of trust. In Stephanidis

(Ed.), HCI international 2020 – Late breaking papers: cognition, learning and games. (Vol. 12425, pp. 202–215). Springer. https://doi.org/10.1007/978-3-030-60128-7_16

58.

Parasuraman

Riley

(1997). Humans and automation: Use, misuse, disuse, abuse. Human Factors, 39(2), 230–253.

59.

Perello-March

Burns

C. G.

Woodman

Elliott

M. T.

Birrell

S. A.

(2023). Using fnirs to verify trust in highly automated driving. IEEE Transactions on Intelligent Transportation Systems. https://doi.org/10.2139/ssrn.4062869

60.

Perello-March

Burns

Woodman

Elliott

M. T.

Birrell

(2021). Driver state monitoring: manipulating reliability expectations in simulated automated driving scenarios. IEEE Transactions on Intelligent Transportation Systems, 1–11. https://doi.org/10.1109/TITS.2021.3050518

61.

Pinti

Scholkmann

Hamilton

Burgess

Tachtsidis

(2019). Current status and issues regarding pre-processing of fNIRS neuroimaging data: An investigation of diverse signal filtering methods within a general linear model framework. Frontiers in Human Neuroscience, 12(January), 1–21. https://doi.org/10.3389/fnhum.2018.00505

62.

Quaresima

Bisconti

Ferrari

(2012). A brief review on the use of functional near-infrared spectroscopy (fNIRS) for language imaging studies in human newborns and adults. Brain and Language, 121(2), 79–89. https://doi.org/10.1016/j.bandl.2011.03.009

63.

Rajaonah

Tricot

Anceaux

Millot

(2008). The role of intervening variables in driver-ACC cooperation. International Journal of Human Computer Studies, 66(3), 185–197. https://doi.org/10.1016/j.ijhcs.2007.09.002

64.

Rangel

Camerer

Montague

(2008). A framework for studying the neurobiology of value-based decision making. Nature Reviews Neuroscience, 9, 545–556. https://doi.org/10.1038/nrn2357

65.

Riley

(1996). Operator Reliance on Automation: Theory and Data. In Mouloua

Raja

(Eds.), Automation and human performance theory and applications (pp. 19–35). Taylor & Francis Group. https://doi.org/10.1201/9781315137957-2

66.

Roche-Labarbe

Zaaimi

Berquin

Nehlig

Grebe

Wallois

(2008). NIRS-measured oxy- and deoxyhemoglobin changes associated with EEG spike-and-wave discharges in children. Epilepsia, 49(11), 1871–1880. https://doi.org/10.1111/j.1528-1167.2008.01711.x

67.

SAE International . (2021). Taxonomy and definitions for terms related to driving automation systems for on-road motor vehicles. https://doi.org/10.4271/J3016_201806

68.

Santosa

Zhai

Fishburn

Huppert

(2018). The NIRS brain analyzir toolbox. Algorithms, 11(5). https://doi.org/10.3390/A11050073

69.

Schroeter

M. L.

Zysset

Kupka

Kruggel

Von Cramon

D. Y.

(2002). Near-infrared spectroscopy can detect brain activity during a color-word matching stroop task in an event-related design. Human Brain Mapping, 17(1), 61–71. https://doi.org/10.1002/hbm.10052

70.

Seet

Harvy

Bose

Dragomir

Bezerianos

Thakor

(2020). Differential impact of autonomous vehicle malfunctions on human trust. IEEE Transactions on Intelligent Transportation Systems, 1–10. https://doi.org/10.1109/tits.2020.3013278

71.

Sibi

Ayaz

Kuhns

D. P.

Sirkin

D. M.

(2016). Monitoring driver cognitive load using functional near infrared spectroscopy in partially autonomous cars. In IEEE Intelligent Vehicles Symposium, Iv.

72.

Singer

Critchley

H. D.

Preuschoff

(2009). A common role of insula in feelings, empathy and uncertainty. Trends in Cognitive Sciences, 13(8), 334–340. https://doi.org/10.1016/j.tics.2009.05.001

73.

Solís-Marcos

Ahlström

Kircher

(2018). Performance of an additional task during level 2 automated driving: An on-road study comparing drivers with and without experience with partial automation. Human Factors, 60(6), 778–792. https://doi.org/10.1177/0018720818773636

74.

Solís-Marcos

Galvao-Carmona

Kircher

(2017). Reduced attention allocation during short periods of partially automated driving: An event-related potentials study. Frontiers in Human Neuroscience, 11, 537. https://doi.org/10.3389/FNHUM.2017.00537/BIBTEX

75.

Taga

Asakawa

Maki

Konishi

Koizumi

(2003). Brain imaging in awake infants by near-infrared optical topography. Proceedings of the National Academy of Sciences of the United States of America, 100(19), 10722–10727. https://doi.org/10.1073/pnas.1932552100

76.

Tak

J. C.

(2014). Statistical analysis of fNIRS data: A comprehensive review. In NeuroImage (Vol. 85, pp. 72–91). https://doi.org/10.1016/j.neuroimage.2013.06.016

77.

Tanida

Sakatani

Takano

Tagai

(2004). Relation between asymmetry of prefrontal cortex activities and the autonomic nervous system during a mental arithmetic task: Near infrared spectroscopy study. Neuroscience Letters, 369(1), 69–74. https://doi.org/10.1016/j.neulet.2004.07.076

78.

Teh

Jamson

Carsten

Jamson

(2014). Temporal fluctuations in driving demand: the effect of traffic complexity on subjective measures of workload and driving performance. Transportation Research Part F: Traffic Psychology and Behaviour, 22, 207–217. https://doi.org/10.1016/j.trf.2013.12.005

79.

Verdière

K. J.

Roy

R. N.

Dehais

(2018). Detecting pilot’s engagement using fNIRS connectivity features in an automated vs. manual landing scenario. Frontiers in Human Neuroscience, 12(January), 1–14. https://doi.org/10.3389/fnhum.2018.00006

80.

Volz

K. G.

Schubotz

R. I.

Von Cramon

D. Y.

(2005). Variants of uncertainty in decision-making and their neural correlates. Brain Research Bulletin, 67(5), 403–412. https://doi.org/10.1016/j.brainresbull.2005.06.011

81.

Wiggins

I. M.

Anderson

C. A.

Kitterick

P. T.

Hartley

D. E. H.

(2016). Speech-evoked activation in adult temporal cortex measured using functional near-infrared spectroscopy (fNIRS): Are the measurements reliable? Hearing Research, 339, 142–154. https://doi.org/10.1016/j.heares.2016.07.007

82.

Wörle

Metz

Thiele

Weller

(2019). Detecting sleep in drivers during highly automated driving: The potential of physiological parameters. IET Intelligent Transport Systems, 13(8), 1241–1248. https://doi.org/10.1049/iet-its.2018.5529

83.

Yücel

M. A.

von Lühmann

Scholkmann

Gervain

Dan

Ayaz

Boas

Cooper

R. J.

Culver

Elwell

C. E.

Eggebrecht

Franceschini

M. A.

Grova

Homae

Lesage

Obrig

Tachtsidis

Tak

Tong

Wolf

(2021). Best Practices for fNIRS publications. Neurophotonics, 8(1), 1–34. https://doi.org/10.1117/1.NPh.8.1.012101

84.

Zimeo Morais

G. A.

Balardin

J. B.

Sato

J. R.

(2018). FNIRS Optodes’ Location Decider (fOLD): A toolbox for probe arrangement guided by brain regions-of-interest. Scientific Reports, 8(1), 1–11. https://doi.org/10.1038/s41598-018-21716-z

How Do Drivers Perceive Risks During Automated Driving Scenarios? An fNIRS Neuroimaging Study

Abstract

Objective

Background

Methods

Results

Conclusion

Application

Keywords

INTRODUCTION

METHOD

Participants

Driving Simulator

Functional Near-Infrared Spectroscopy

Questionnaires

Experimental Procedure

Data Preprocessing

Data Analysis

RESULTS

Hypothesis 1 – Perceived Risk From Slowly Evolving Conditions

Hypothesis 2 – Perceived Risk From Rapidly Evolving Conditions

Psychometric Results

DISCUSSION

Hypothesis 1

Hypothesis 2

Limitations and Future Work

CONCLUSION

KEY POINTS

Footnotes

Acknowledgements

ORCID iDs

References