This study developed a fixation-related electroencephalography band power (FRBP) approach for situation awareness (SA) assessment in automated driving.
Background
Maintaining good SA in Level 3 automated vehicles is crucial to drivers' takeover performance when the automated system fails. A multimodal fusion approach that enables the analysis of the visual behavioral and cognitive processes of SA can facilitate real-time assessment of SA in future driver state monitoring systems.
Method
Thirty participants performed three simulated automated driving tasks. After each task, the Situation Awareness Global Assessment Technique (SAGAT) was deployed to capture their SA about key elements that could affect their takeover task performance. Participants eye movements and brain activities were recorded. Data on their brain activity after each eye fixation on the key elements were extracted and labeled according to the correctness of the SAGAT. Mixed-effects models were used to identify brain regions that were indicative of SA, and machine learning models for SA assessment were developed based on the identified brain regions.
Results
Participants’ alpha and theta oscillation at frontal and temporal areas are indicative of SA. In addition, the FRBP technique can be used to predict drivers’ SA with an accuracy of 88% using a neural network model.
Conclusion
The FRBP technique, which incorporates eye movements and brain activities, can provide more comprehensive evaluation of SA. Findings highlight the potential of utilizing FRBP to monitor drivers’ SA in real-time.
Application
The proposed framework can be expanded and applied to driver state monitoring systems to measure human SA in real-world driving.
AbbassH. A.TangJ.AminR.EllejmiM.KirbyS. (2014). Augmented cognition using real-time EEG-based adaptive strategies for air traffic control. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 58(1), 230–234. https://doi.org/10.1177/1541931214581048
2.
Bigdely-ShamloN.MullenT.KotheC.SuK. M.RobbinsK. A. (2015). The PREP pipeline: Standardized preprocessing for large-scale EEG analysis. Frontiers in Neuroinformatics, 9(JUNE), 16–19. https://doi.org/10.3389/fninf.2015.00016
3.
BoisgueheneucF. duLevyR.VolleE.SeassauM.DuffauH.KinkingnehunS.SamsonY.ZhangS.DuboisB. (2006). Functions of the left superior frontal gyrus in humans: A lesion study. Brain: A Journal of Neurology, 129(Pt 12), 3315–3328. https://doi.org/10.1093/brain/awl244
4.
BurkeJ. F.ZaghloulK. A.JacobsJ.WilliamsR. B.SperlingM. R.SharanA. D.KahanaM. J. (2013). Synchronous and asynchronous theta and gamma activity during episodic memory formation. Journal of Neuroscience: The Official Journal of the Society for Neuroscience, 33(1), 292–304. https://doi.org/10.1523/JNEUROSCI.2057-12.2013
5.
CatherwoodD.EdgarG. K.NikollaD.AlfordC.BrookesD.BakerS.WhiteS. (2014). Mapping brain activity during loss of situation awareness: An EEG investigation of a basis for top-down influence on perception. Human Factors, 56(8), 1428–1452. https://doi.org/10.1177/0018720814537070
6.
ChenH.LiuS.PangL.WanyanX.FangY. (2021). Developing an improved ACT-R model for pilot situation awareness measurement. IEEE Access, 9, 122113–122124. https://doi.org/10.1109/ACCESS.2021.3108438
7.
ClarkeD. D.WardP.BartleC.TrumanW. (2007). The role of motorcyclist and other driver behaviour in two types of serious accident in the UK. Accident; Analysis and Prevention, 39(5), 974–981. https://doi.org/10.1016/j.aap.2007.01.002
8.
CoullJ. T.FrithC. D.FrackowiakR. S. J.GrasbyP. M. (1996). A fronto-parietal network for rapid visual information processing: A PET study of sustained attention and working memory. Neuropsychologia, 34(11), 1085–1095. https://doi.org/10.1016/0028-3932(96)00029-2
9.
DelormeA.MakeigS. (2004). EEGLAB: An open source toolbox for analysis of single-trial EEG dynamics including independent component analysis. Journal of Neuroscience Methods, 134(1), 9–21. https://doi.org/10.1016/j.jneumeth.2003.10.009
10.
de WinterJ. C. F.EismaY. B.CabrallC. D. D.HancockP. A.StantonN. A. (2019). Situation awareness based on eye movements in relation to the task environment. Cognition, Technology & Work, 21(1), 99–111. https://doi.org/10.1007/s10111-018-0527-6
11.
DelormeAMakeigS (2004). EEGLAB: an open-source toolbox for analysis of single-trial EEG dynamics. Journal of Neuroscience Methods, 134, 9–21.
EndsleyM. R. (1995b). Measurement of situation awareness in dynamic systems. Human Factors: The Journal of the Human Factors and Ergonomics Society, 37(1), 65–84. https://doi.org/10.1518/001872095779049499
14.
EndsleyM. R. (2021). A systematic review and meta-analysis of direct objective measures of situation awareness: A comparison of SAGAT and SPAM. Human Factors, 63(1), 124–150. https://doi.org/10.1177/0018720819875376
15.
EndsleyM. R.RobertsonM. M. (1996). Team situation awareness in aviation maintenance. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 40(21), 1077–1081. https://doi.org/10.1177/154193129604002107
16.
Fernandez RojasR.DebieE.FidockJ.BarlowM.KasmarikK.AnavattiS.GarrattM.AbbassH. (2019). Encephalographic Assessment of Situation Awareness in Teleoperation of Human-Swarm Teaming. Communications in Computer and Information Science. Communications in Computer and Information Science, 530–539. https://doi.org/10.1007/978-3-030-36808-1_58
17.
FreyA.LemaireB.VercueilL.Guérin-DuguéA. (2018). An eye fixation-related potential study in two reading tasks: Reading to memorize and reading to make a decision. Brain Topography, 31(4), 640–660. https://doi.org/10.1007/s10548-018-0629-8
18.
GianniniM.AlexanderD. M.NikolaevA. R.van LeeuwenC. (2018). Large-scale traveling waves in EEG activity following eye movement. Brain Topography, 31(4), 608–622. https://doi.org/10.1007/s10548-018-0622-2
19.
GraupnerS. T.PannaschS.VelichkovskyB. M. (2011). Saccadic context indicates information processing within visual fixations: Evidence from event-related potentials and eye-movements analysis of the distractor effect. International Journal of Psychophysiology: Official Journal of the International Organization of Psychophysiology, 80(1), 54–62. https://doi.org/10.1016/J.IJPSYCHO.2011.01.013
20.
GrippenkovenJ.DietschS. (2016). Gaze direction and driving behavior of drivers at level crossings. Journal of Transportation Safety & Security, 8(sup1), 4–18. https://doi.org/10.1080/19439962.2015.1046620
21.
HasanzadehS.EsmaeiliB.DoddM. D. (2018). Examining the relationship between construction workers’ visual attention and situation awareness under fall and tripping hazard conditions: Using mobile eye tracking. Journal of Construction Engineering and Management, 144(7). https://doi.org/10.1061/(ASCE)CO.1943-7862.0001516
22.
HergethS.LorenzL.KremsJ. F.ToenertL. (2015). Effects of take-over requests and cultural background on automation trust in highly automated driving (pp. 331–337). https://doi.org/10.17077/drivingassessment.1591
23.
HerrmannC. S.RachS.VosskuhlJ.StrüberD. (2014). Time-frequency analysis of event-related potentials: A brief tutorial. Brain Topography, 27(4), 438–450. https://doi.org/10.1007/s10548-013-0327-5
24.
HillaY.von MankowskiJ.FöckerJ.SausengP. (2020). Faster visual information processing in video gamers is associated with EEG alpha amplitude modulation. Frontiers in Psychology, 11, 599788. https://doi.org/10.3389/fpsyg.2020.599788
25.
KaurA.ChaujarR.ChinnaduraiV. (2019). Effects of neural mechanisms of pretask resting EEG alpha information on situational awareness: A functional connectivity approach. Human Factors, 62(7), 1150–1170. https://doi.org/10.1177/0018720819869129
26.
KlimeschW. (1999). EEG alpha and theta oscillations reflect cognitive and memory performance: A review and analysis. Brain Research. Brain Research Reviews, 29(2–3), 169–195. https://doi.org/10.1016/S0165-0173(98)00056-3
27.
KlimeschW. (2012). Alpha-band oscillations, attention, and controlled access to stored information. Trends in Cognitive Sciences, 16(12), 606–617. https://doi.org/10.1016/j.tics.2012.10.007
28.
LegaB. C.JacobsJ.KahanaM. (2012). Human hippocampal theta oscillations and the formation of episodic memories. Hippocampus, 22(4), 748–761. https://doi.org/10.1002/HIPO.20937
29.
LiangN.YangJ.YuD.Prakah-AsanteK. O.CurryR.BlommerM.SwaminathanR.PittsB. J. (2021). Using eye-tracking to investigate the effects of pre-takeover visual engagement on situation awareness during automated driving. Accident; analysis and prevention, 157, 106143. https://doi.org/10.1016/j.aap.2021.106143
30.
LiashchynskyiP. (2019). Grid search, random search, genetic algorithm: A big comparison for NAS. http://arxiv.org/abs/1912.06059
MohamadS. M.RazaliA. (2019). An eye tracking analysis on diagnostic performance of digital fundus photography images between ophthalmologists and optometrists. International Journal of Environmental Research and Public Health, 17(1), 30. https://doi.org/10.3390/ijerph17010030
34.
PattenC. J. D.KircherA.ÖstlundJ.NilssonL.SvensonO. (2006). Driver experience and cognitive workload in different traffic environments. Accident Analysis & Prevention, 38(5), 887–894. https://doi.org/10.1016/j.aap.2006.02.014
35.
PetersenP.VoigtlaenderF. (2018). Optimal approximation of piecewise smooth functions using deep ReLU neural networks. Neural Networks, 108, 296–330. https://doi.org/10.1016/j.neunet.2018.08.019
36.
PudjihartonoN.FadasonT.Kempa-LiehrA. W.O’SullivanJ. M. (2022). A review of feature selection methods for machine learning-based disease risk prediction. Frontiers in Bioinformatics, 2, 927312. https://doi.org/10.3389/fbinf.2022.927312
RiesA. J.TouryanJ.AhrensB.ConnollyP. (2016). The impact of task demands on fixation-related brain potentials during guided search. PLoS One, 11(6), 01572600. https://doi.org/10.1371/journal.pone.0157260
39.
RoachB. J.MathalonD. H. (2008). Event-related EEG time-frequency analysis: An overview of measures and an analysis of early gamma band phase locking in schizophrenia. Schizophrenia Bulletin, 34(5), 907–926. https://doi.org/10.1093/schbul/sbn093
ScharingerC. (2018). Fixation-related EEG frequency band power analysis: A promising methodology for studying instructional design effects of multimedia learning material. Frontline Learning Research, 6(3), 57–71. https://doi.org/10.14786/flr.v6i3.373
42.
ScholzS.SchneiderS. L.RoseM. (2017). Differential effects of ongoing EEG beta and theta power on memory formation. PLoS One, 12(2), Article e0171913. https://doi.org/10.1371/JOURNAL.PONE.0171913
43.
SonJ.ParkM. (2017). Situation awareness and transitions in highly automated driving: A framework and mini review. Journal of Ergonomics, 07(05), 1–6. https://doi.org/10.4172/2165-7556.1000212
44.
StaudiglT.HanslmayrS. (2013). Theta oscillations at encoding mediate the context-dependent nature of human episodic memory. Current Biology: CB, 23(12), 1101–1106. https://doi.org/10.1016/J.CUB.2013.04.074
45.
TakedaY.SugaiM.YagiA. (2001). Eye fixation related potentials in a proof reading task. International Journal of Psychophysiology: Official Journal of the International Organization of Psychophysiology, 40(3), 181–186. https://doi.org/10.1016/s0167-8760(00)00185-9
46.
TrapsilawatiF.HerliansyahM. K.NugraheniA. S. A. N. S.FatikasariM. P.TissamodieG. (2020). EEG-based analysis of air traffic conflict: Investigating controllers’ situation awareness, stress level and brain activity during conflict resolution. Journal of Navigation, 73(3), 678–696. https://doi.org/10.1017/S0373463319000882
47.
VignaliL.HimmelstossN. A.HawelkaS.RichlanF.HutzlerF. (2016). Oscillatory brain dynamics during sentence reading: A fixation-related spectral perturbation analysis. Frontiers in Human Neuroscience, 10(April), 191–213. https://doi.org/10.3389/fnhum.2016.00191
48.
WilschutE.HoogeboomP.MulderB.HansonE.WijersB. (2005). Effects of workload and time-on-task effects on eye fixation related brain potentials in a simulated air traffic control task. Human Factors in Design, Safety, and Management, 2005(0), 1–13. https://doi.org/10.1016/j.apergo.2019.01.008
49.
YangJ.LiangN.PittsB. J.Prakah-AsanteK. O.CurryR.BlommerM.SwaminathanR.YuD. (2023). Multimodal sensing and computational intelligence for situation awareness classification in autonomous driving. IEEE Transactions on Human-Machine Systems, 53(2), 270–281. https://doi.org/10.1109/THMS.2023.3234429
50.
YeoL. G.SunH.LiuY.TrapsilawatiF.SourinaO.ChenC. H.AngW. T. (2017). Mobile EEG- based situation awareness. In 2017 IEEE International Conference on Systems, Man, and Cybernetics, SMC 2017 (pp. 3030–3035), Banff, AB, Canada, 2017.
51.
YotoA.KatsuuraT.IwanagaK.ShimomuraY. (2007). Effects of object color stimuli on human brain activities in perception and attention referred to EEG alpha band response. Journal of Physiological Anthropology, 26(3), 373–379. https://doi.org/10.2114/jpa2.26.373
52.
YuvarajR.LyeS. W.WeeH. J. (2020). A real time neurophysiological framework for general monitoring awareness of air traffic controllers. In 2020 IEEE Asia-Pacific Conference on Computer Science and Data Engineering (CSDE), Gold Coast, Australia (pp. 1–4). https://doi.org/10.1109/CSDE50874.2020.9411589
53.
ZhangT.YangJ.LiangN.PittsB. J.Prakah-AsanteK.CurryR.DuerstockB.WachsJ. P.YuD. (2021). Physiological measurements of situation awareness: A systematic review. Human factors, 65(5), 737–758. https://doi.org/10.1177/0018720820969071
54.
ZhaoN.ChenW.XuanY.MehlerB.ReimerB.FuX. (2014). Drivers’ and non-drivers’ performance in a change detection task with static driving scenes: Is there a benefit of experience?Ergonomics, 57(7), 998–1007. https://doi.org/10.1080/00140139.2014.909952
55.
ZhouF.YangX. J.De WinterJ. C. F. (2022). Using eye-tracking data to predict situation awareness in real time during takeover transitions in conditionally automated driving. IEEE Transactions on Intelligent Transportation Systems, 23(3), 2284–2295. https://doi.org/10.1109/tits.2021.3069776
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
