This study aims to investigate the causes of take-over failures in conditional automated driving with spatial-temporal analysis of brain zone activation.
Background
Take-over requires a human driver to resume the control of the vehicle when its automation system disengages. Existing studies have found that take-over failures occur frequently on some drivers, but the causes have not been thoroughly studied.
Method
In a driving simulator experiment, 40 drivers took over in critical freeway cut-in situations. Functional near-infrared spectroscopy (fNIRS) data were collected before and during the take-over process to evaluate brain zone activation. Successful and failed take-overs were compared with changes in fNIRS data based on spatial-temporal comparisons and cluster analysis.
Results
The results suggested a significant difference in temporal brain activation between take-over failure and success conditions. Take-over failure conditions are mostly related to earlier and longer brain activation in most brain zones and repeated activation of the cognition brain zones. Drivers’ attention switches, steering, and braking patterns are also related to different brain zone activation orders.
Conclusion
The results indicate the need to reduce the mental workload caused by the sudden system disengagement to prevent take-over failure.
Application
Future research and implementation should focus on earlier warnings of upcoming hazards and driver education in dealing with sudden system disengagement.
ArakawaT.HibiR.FujishiroT. (2019). Psychophysical assessment of a driver’s mental state in autonomous vehicles. Transportation Research Part A: Policy and Practice, 124, 587–610. https://doi.org/10.1016/j.tra.2018.05.003
2.
BaltersS.BakerJ. M.GeesemanJ. W.ReissA. L. (2021). A methodological review of fNIRS in driving research: Relevance to the future of autonomous vehicles. Frontiers in Human Neuroscience, 15, 637589. https://doi.org/10.3389/fnhum.2021.637589
BroadbentD. P.D’InnocenzoG.EllmersT. J.ParslerJ.SzameitatA. J.BishopD. T. (2023). Cognitive load, working memory capacity and driving performance: A preliminary fNIRS and eye tracking study. Transportation Research Part F: Traffic Psychology and Behaviour, 92, 121–132. https://doi.org/10.1016/j.trf.2022.11.013
5.
CarvalhoK.PearlsonG.AsturR.CalhounV. (2006). Simulated driving and brain imaging: Combining behavior, brain activity, and virtual reality. CNS Spectrums, 11(1), 52–62. https://doi.org/10.1017/s1092852900024214
6.
ChoiD.SatoT.AndoT.AbeT.AkamatsuM.KitazakiS. (2020). Effects of cognitive and visual loads on driving performance after take-over request (TOR) in automated driving. Applied Ergonomics, 85, 103074. https://doi.org/10.1016/j.apergo.2020.103074
7.
CrundallD.ChapmanP.TrawleyS.CollinsL.Van LoonE.AndrewsB.UnderwoodG. (2012). Some hazards are more attractive than others: Drivers of varying experience respond differently to different types of hazard. Accident Analysis & Prevention, 45, 600–609. https://doi.org/10.1016/j.aap.2011.09.049
8.
CuiX.BrayS.BryantD. M.GloverG. H.ReissA. L. (2011). A quantitative comparison of nirs and fMRI across multiple cognitive tasks. NeuroImage, 54(4), 2808–2821. https://doi.org/10.1016/j.neuroimage.2010.10.069
9.
DoganE.HonnêtV.MasfrandS.GuillaumeA. (2019). Effects of non-driving-related tasks on take-over performance in different take-over situations in conditionally automated driving. Transportation Research Part F: Traffic Psychology and Behaviour, 62, 494–504. https://doi.org/10.1016/j.trf.2019.02.010
10.
DuN.YangX. J.ZhouF. (2020). Psychophysiological responses to take-over requests in conditionally automated driving. Accident Analysis & Prevention, 148, 105804. https://doi.org/10.1016/j.aap.2020.105804
11.
DurantinG.GagnonJ. F.TremblayS.DehaisF. (2014). Using near-infrared spectroscopy and heart rate variability to detect mental overload. Behavioural Brain Research, 259, 16–23. https://doi.org/10.1016/j.bbr.2013.10.042
12.
EbnaliM.LambR.FathiR.HulmeK. (2021). Virtual reality tour for first-time users of highly automated cars: Comparing the effects of virtual environments with different levels of interaction fidelity. Applied Ergonomics, 90, 103226. https://doi.org/10.1016/j.apergo.2020.103226
13.
ErikssonA.,StantonN. A. (2017). Take-over time in highly automated vehicles: Noncritical transitions to and from manual control. Human Factors, 59(4), 689–705. https://doi.org/10.1177/0018720816685832
14.
FeldhuetterA.RuhlA.FeierleA.BenglerK. (2019). The effect of fatigue on take-over performance in urgent situations in conditionally automated driving. Proceedings of IEEE intelligent transportation system conference (pp. 1889–1894). IEEE.
15.
FoyH. J.ChapmanP. (2018). Mental workload is reflected in driver behaviour, physiology, eye movements and prefrontal cortex activation. Applied Ergonomics, 73, 90–99. https://doi.org/10.1016/j.apergo.2018.06.006
16.
GençE.SchlüterC.FraenzC.ArningL.MetzenD.NguyenH. P.VoelkleM. C.StreitF.GüntürkünO.KumstaR.OcklenburgS. (2021). Polygenic scores for cognitive abilities and their association with different aspects of general intelligence—A deep phenotyping approach. Molecular Neurobiology, 58(8), 4145–4156. https://doi.org/10.1007/s12035-021-02398-7
17.
GrudenT.TomažičS.SodnikJ.JakusG. (2022). A user study of directional tactile and auditory user interfaces for take-over requests in conditionally automated vehicles. Accident Analysis & Prevention, 174, 106766. https://doi.org/10.1016/j.aap.2022.106766
18.
HaghaniM.BliemerM. C. J.FarooqB.KimI.LiZ.OhC.ShahhoseiniZ.MacDougallH. (2021). Applications of brain imaging methods in driving behaviour research. Accident Analysis & Prevention, 154, 106093. https://doi.org/10.1016/j.aap.2021.106093
19.
HeroldF.WiegelP.ScholkmannF.ThiersA.HamacherD.SchegaL. (2017). Functional near-infrared spectroscopy in movement science: A systematic review on cortical activity in postural and walking tasks. Neurophotonics, 4(4), 041403. https://doi.org/10.1117/1.NPh.4.4.041403
20.
HuangG.PittsB. J. (2022). The effects of age and physical exercise on multimodal signal responses: Implications for semi-autonomous vehicle take-over requests. Applied Ergonomics, 98, 103595. https://doi.org/10.1016/j.apergo.2021.103595
LiC.LiX.LvM.ChenF.MaX.ZhangL. (2022). How does approaching a lead vehicle and monitoring request affect drivers’ take-over performance? A simulated driving study with functional MRI. International Journal of Environmental Research and Public Health, 19(1), 412. https://doi.org/10.3390/ijerph19010412
23.
LiX.RakotonirainyA.YanX. (2019). How do drivers avoid collisions? A driving simulator-based study. Journal of Safety Research, 70, 89–96. https://doi.org/10.1016/j.jsr.2019.05.002
24.
LiaoP.SunX.ZhangD. (2021). A multimodal study to measure the cognitive demands of hazard recognition in construction workplaces. Safety Science, 133, 105010. https://doi.org/10.1016/j.ssci.2020.105010
25.
LiuZ.ZhangM.XuG.HuoC.TanQ.LiZ.YuanQ. (2017). Effective connectivity analysis of the brain network in drivers during actual driving using near-infrared spectroscopy. Frontiers in Behavioral Neuroscience, 11(11), 00211. https://doi.org/10.3389/fnbeh.2017.00211
26.
LouwT.MarkkulaG.BoerE.MadiganR.CarstenO.MeratN. (2017). Coming back into the loop: Drivers’ perceptual-motor performance in critical events after automated driving. Accident Analysis & Prevention, 108, 9–18. https://doi.org/10.1016/j.aap.2017.08.011
27.
MalhotraP.CoulthardE. J.HusainM. (2009). Role of right posterior parietal cortex in maintaining attention to spatial locations over time. Brain: A Journal of Neurology, 132(Pt 3), 645–660. https://doi.org/10.1093/brain/awn350
28.
McdonaldA. D.AlambeigiH.EngstromJ.MarkkulaG.VogelpohlT.DunneJ.YumaN. (2019). Toward computational simulations of behavior during automated driving take-overs: A review of the empirical and modeling literature. Human Factors, 61(4), 642–688. https://doi.org/10.1177/0018720819829572
29.
MedagliaJ.ZurnP.Sinnott-ArmstrongW.BassettD. S. (2017). Mind control as a guide for the mind. Nature Human Behaviour, 1(6), 0119. https://doi.org/10.1038/s41562-017-0119
30.
NilssonE. J.BärgmanJ.Ljung AustM.MatthewsG.SvanbergB. (2022). Let complexity bring clarity: A multidimensional assessment of cognitive load using physiological measures. Frontiers in Neuroergonomics, 3, 787295. https://doi.org/10.3389/fnrgo.2022.787295
31.
PetridesM.,PandyaD. N. (2007). Efferent association pathways from the rostral prefrontal cortex in the macaque monkey. Journal of Neuroscience: The Official Journal of the Society for Neuroscience, 27(43), 11573–11586. https://doi.org/10.1523/JNEUROSCI.2419-07.2007
32.
ProverbioA. M.SantoniS.AdorniR. (2020). ERP markers of valence coding in emotional speech processing. iScience, 23(3), 100933. https://doi.org/10.1016/j.isci.2020.100933
33.
Ray SarkarD.Ramachandra RaoK.ChatterjeeN. (2024). A review of surrogate safety measures on road safety at unsignalized intersections in developing countries. Accident Analysis & Prevention, 195, 107380. https://doi.org/10.1016/j.aap.2023.107380
34.
RocheF.BeckerS.ThüringM. (2022). What happens when drivers of automated vehicles take over control in critical lane change situations?Transportation Research Part F: Traffic Psychology and Behaviour, 84, 407–4422. https://doi.org/10.1016/j.trf.2021.11.021
35.
SAE International. (2018). Sae J3016: Taxonomy and definitions for terms related to driving automation systems for on-road motor vehicles J3016. SAE International.
36.
ScholkmannF.KleiserS.MetzA. J.ZimmermannR.Mata PaviaJ.WolfU.WolfM. (2014). A review on continuous wave functional near-infrared spectroscopy and imaging instrumentation and methodology. NeuroImage, 85(SI Pt 1), 6–27. https://doi.org/10.1016/j.neuroimage.2013.05.004
37.
SeetM.DragomirA.HarvyJ.ThakorN. V.BezerianosA. (2022). Objective assessment of trait attentional control predicts driver response to emergency failures of vehicular automation. Accident Analysis & Prevention, 168, 106588. https://doi.org/10.1016/j.aap.2022.106588
38.
ShahiniF.ZahabiM. (2022). Effects of levels of automation and non-driving related tasks on driver performance and workload: A review of literature and meta-analysis. Applied Ergonomics, 104, 103824. https://doi.org/10.1016/j.apergo.2022.103824
39.
SitaramR.RosT.StoeckelL.HallerS.ScharnowskiF.Lewis-PeacockJ.WeiskopfN.BlefariM. L.RanaM.OblakE.BirbaumerN.SulzerJ. (2017). Closed-loop brain training: The science of neurofeedback. Nature Reviews Neuroscience, 18(2), 86–100. https://doi.org/10.1038/nrn.2016.164
UnniA.TrendeA.PauleyC.WeberL.BieblB.KaciankaS.LüdtkeA.BenglerK.PretschnerA.FränzleM.RiegerJ. W. (2022). Investigating differences in behavior and brain in human-human and human-autonomous vehicle interactions in time-critical situations. Frontiers in Neuroergonomics, 3, 836518. https://doi.org/10.3389/fnrgo.2022.836518
43.
WalchM.LangeK.BaumannM.WeberM. (2017). Autonomous driving: Investigating the feasibility of car-driver handover assistance. International Journal of Mobile Human Computer Interaction, 9(2), 58–74. https://doi.org/10.4018/IJMHCI.2017040104
44.
WangC.XieY.HuangH.LiuP. (2021). A review of surrogate safety measures and their applications in connected and automated vehicles safety modeling. Accident Analysis & Prevention, 157, 106157. https://doi.org/10.1016/j.aap.2021.106157
45.
WangX.YangM.HurwitzD. (2019). Analysis of cut-in behavior based on naturalistic driving data. Accident Analysis & Prevention, 124, 127–137. https://doi.org/10.1016/j.aap.2019.01.006
46.
WangY.ZhangW.ZhouR. (2022). Speech-based take-over requests in conditionally automated driving: Effects of different voices on the driver take-over performance. Applied Ergonomics, 101, 103695. https://doi.org/10.1016/j.apergo.2022.103695
47.
WeaverB. W.,DeluciaP. R. (2022). A systematic review and meta-analysis of take-over performance during conditionally automated driving. Human Factors, 64(7), 1227–1260. https://doi.org/10.1177/0018720820976476
48.
WuB.FengZ.ZhuX.MaZ. (2019). Analysis of boundary condition model under cut-in scenarios based on logistical regression method. SAE Technical Paper. https://doi.org/10.4271/2019-01-5047
49.
WuK.LinY. (2019). Exploring the effects of critical driving situations on driver perception time (PT) using SHRP2 naturalistic driving study data. Accident Analysis & Prevention, 128, 94–102. https://doi.org/10.1016/j.aap.2019.04.003
50.
YamamotoK.TakahashiH.SugimachiT.NakanoK.SudaY.KatoT. (2018). The study of driver’s brain activity and behavior using fNIRS during actual car driving. Proceedings of the 2018 11th international conference on Human System Interaction (HSI) (pp. 418–424). IEEE.
51.
YoshinoK.OkaN.YamamotoK.TakahashiH.KatoT. (2013). Correlation of prefrontal cortical activation with changing vehicle speeds in actual driving: A vector-based functional near-infrared spectroscopy study. Frontiers in Human Neuroscience, 7, 895. https://doi.org/10.3389/fnhum.2013.00895
52.
ZeebK.BuchnerA.SchraufM. (2015). What determines the take-over time? An integrated model approach of driver take-over after automated driving. Accident Analysis & Prevention, 78, 212–221. https://doi.org/10.1016/j.aap.2015.02.023
53.
ZhangB.de WinterJ.VarottoS.HappeeR.MartensM. (2019). Determinants of take-over time from automated driving: A meta-analysis of 129 studies. Transportation Research Part F: Traffic Psychology and Behaviour, 64, 285–307. https://doi.org/10.1016/j.trf.2019.04.020
54.
ZhouX.HuY.LiaoP.ZhangD. (2021). Hazard differentiation embedded in the brain: A near-infrared spectroscopy-based study. Automation in Construction, 122, 103473. https://doi.org/10.1016/j.autcon.2020.103473
