This study explores the effectiveness of conversational prompts on enhancing driver monitoring behavior and takeover performance in partially automated driving under two non-driving-related task (NDRT) scenarios with varying workloads.
Background
Driver disengagement in partially automated driving is a serious safety concern. Intermittent conversational prompts that require responses may be a solution. However, existing literature is limited with inconsistent findings. There is little consideration of NDRTs as an important context, despite their ubiquitous involvement. A method is also lacking to measure driver engagement at the cognitive level, beyond manual and visual engagements.
Methods
Participants operated a partially automated vehicle in a simulator across six predefined drives. In each drive, participants either received driving-related prompts, daily-conversation prompts, or no prompts, with or without a takeover notification. The first experiment instructed participants to engage in NDRTs at their choice and the second experiment incentivized solving demanding anagrams with monetary rewards.
Results
When participants were voluntarily engaged in NDRTs, answering driving-related prompts and receiving takeover notifications improved their monitoring behavior and takeover performance. However, when participants were involved in the more demanding and incentivized NDRT, answering prompts had little effect.
Conclusion
The study supports the importance of both maintaining appropriate workload and processing driving-related information during partially automated driving. Driving-related prompts improve driver engagement and takeover performance, but they are not robust enough to compete with NDRTs that have high motivational appeals and cognitive demands.
Application
The design of driver engagement tools should consider the workload and information processing mechanisms.
BaiX.FengJ. (2024). Capturing the mind: Non-driving-related tasks as a window into cognitive engagement in automated driving. Proceedings of the Human Factors and Ergonomics Society - Annual Meeting, 68(1), 1362–1364. https://doi.org/10.1177/10711813241262273
3.
BanksV. A.ErikssonA.O’DonoghueJ.StantonN. A. (2018). Is partially automated driving a bad idea? Observations from an on-road study. Applied Ergonomics, 68, 138–145. https://doi.org/10.1016/j.apergo.2017.11.010
4.
BiondiF. N.LohaniM.HopmanR.MillsS.CooperJ. M.StrayerD. L. (2018). 80 MPH and out-of-the-loop: Effects of real-world semi-automated driving on driver workload and arousal. In Proceedings of the Human Factors and Ergonomics Society - Annual Meeting (Vol. 62, pp. 1878–1882). Sage Publications. https://doi.org/10.1177/1541931218621427
5.
BoosA.FeldhütterA.SchwiebacherJ.BenglerK. (2020). Mode errors and intentional violations in visual monitoring of level 2 driving automation. In 2020 IEEE 23rd international conference on intelligent transportation systems (ITSC) (pp. 1–7). IEEE.
BrykA.RaudenbushS. W. (1992). Hierarchical linear models for social and behavioral research: Applications and data analysis methods. Sage.
8.
CabrallC. D.ErikssonA.DregerF.HappeeR.de WinterJ. (2019). How to keep drivers engaged while supervising driving automation? A literature survey and categorisation of six solution areas. Theoretical Issues in Ergonomics Science, 20(3), 332–365. https://doi.org/10.1080/1463922x.2018.1528484
ClarkH.FengJ. (2017). Age differences in the takeover of vehicle control and engagement in non-driving-related activities in simulated driving with conditional automation. Accident Analysis & Prevention, 106, 468–479. https://doi.org/10.1016/j.aap.2016.08.027
11.
da SilvaF. P. (2014). Mental workload, task demand and driving performance: What relation?Procedia - Social and Behavioral Sciences, 162(1), 310–319. https://doi.org/10.1016/j.sbspro.2014.12.212
12.
DeGuzmanC. A.KanaanD.HopkinsS. A.DonmezB. (2021). Takeover request (TOR) effects during different automated vehicle failures. Journal of Intelligent Transportation Systems, 1–31. https://doi.org/10.1080/15472450.2021.1891536
13.
De WinterJ. C.HappeeR.MartensM. H.StantonN. A. (2014). Effects of adaptive cruise control and highly automated driving on workload and situation awareness: A review of the empirical evidence. Transportation Research Part F: Traffic Psychology and Behaviour, 27, 196–217. https://doi.org/10.1016/j.trf.2014.06.016
14.
DoganE.RahalM. C.DeborneR.DelhommeP.KemenyA.PerrinJ. (2017). Transition of control in a partially automated vehicle: Effects of anticipation and non-driving-related task involvement. Transportation Research Part F: Traffic Psychology and Behaviour, 46, 205–215. https://doi.org/10.1016/j.trf.2017.01.012
15.
ErikssonA.StantonN. A. (2017). Takeover time in highly automated vehicles: Noncritical transitions to and from manual control. Human Factors, 59(4), 689–705. https://doi.org/10.1177/0018720816685832
16.
ForsterY.GeiselV.HergethS.NaujoksF.KeinathA. (2020). Engagement in non-driving related tasks as a non-intrusive measure for mode awareness: A simulator study. Information, 11(5), 239. https://doi.org/10.3390/info11050239
17.
GedenM.StaicuA.FengJ. (2018). The impacts of perceptual load and driving duration on mind wandering in driving. Transportation Research Part F: Traffic Psychology and Behaviour, 57, 75–83. https://doi.org/10.1016/j.trf.2017.07.004
18.
GreenleeE. T.DeLuciaP. R.NewtonD. C. (2018). Driver vigilance in automated vehicles: Hazard detection failures are a matter of time. Human Factors, 60(4), 465–476. https://doi.org/10.1177/0018720818761711
19.
HalinA.VerlyJ. G.DroogenbroeckM. V. (2021). Survey and synthesis of state of the art in driving monitoring. Sensors, 21(16), 5558. https://doi.org/10.48550/arXiv.2110.00472
HartS. G.StavelandL. E. (1988). Development of NASA-TLX (Task Load Index): Results of empirical and theoretical research. Advances in Psychology, 52, 139–183. https://doi.org/10.1016/S0166-4115(08)62386-9
22.
HoffmanL.RovineM. J. (2007). Multilevel models for the experimental psychologist: Foundations and illustrative examples. Behavior Research Methods, 39(1), 101–117. https://doi.org/10.3758/bf03192848
23.
JausseinM.LévêqueL.DenielJ.BelletT.TattegrainH.Marin-LamelletC. (2021). How do non-driving-related tasks affect engagement under highly automated driving situations? A literature review. Frontiers in Future Transportation, 2, Article 687602. https://doi.org/10.3389/ffutr.2021.687602
24.
JavidR.SadeghvaziriE.JeihaniM. (2023). A driving simulator study to understand the impact of cell phone blocking apps on distraction. Highlights of Vehicles, 1(1), 17–28. https://doi.org/10.54175/hveh1010002
25.
KahnemanD. (1973). Attention and effort (Vol. 1063, pp. 218–226). Prentice-Hall.
26.
KesslerR. C.AdlerL.AmesM.DemlerO.FaraoneS.HiripiE. V. A.HowesM. J.JinR.SecnikK.SpencerT.UstunT. B.WaltersE. E. (2005). The World Health Organization Adult ADHD Self-Report Scale (ASRS): A short screening scale for use in the general population. Psychological Medicine, 35(2), 245–256. https://doi.org/10.1017/s0033291704002892
27.
KörberM.CingelA.ZimmermannM.BenglerK. (2015). Vigilance decrement and passive fatigue caused by monotony in automated driving. Procedia Manufacturing, 3, 2403–2409. https://doi.org/10.1016/j.promfg.2015.07.499
28.
LeeJ.HiranoT.HanoT.ItohM. (2019). Conversation during partially automated driving: How attention arousal is effective on maintaining situation awareness. In 2019 IEEE international conference on systems, man and cybernetics (SMC) (pp. 3718–3723). IEEE.
29.
LiX.SchroeterR.RakotonirainyA.KuoJ.LennéM. G. (2023). Get ready for take-overs: Using head-up display for drivers to engage in non–driving-related tasks in automated vehicles. Human Factors, 65(8), 1759–1775. https://doi.org/10.1177/00187208211056200
30.
LouwT.MeratN. (2017). Are you in the loop? Using gaze dispersion to understand driver visual attention during vehicle automation. Transportation Research Part C: Emerging Technologies, 76, 35–50. https://doi.org/10.1016/j.trc.2017.01.001
31.
MaS.ZhangW.YangZ.KangC.WuC.ChaiC.ShiJ.LiH. (2020). Promote or inhibit: An inverted U-shaped effect of workload on driver takeover performance. Traffic Injury Prevention, 21(7), 482–487. https://doi.org/10.1080/15389588.2020.1804060
32.
MagnussenA. F.LeN.HuL.Eric WongW. (2020). A survey of the inadequacies in traffic sign recognition systems for autonomous vehicles. International Journal of Performability Engineering, 16(10), 1588–1597. https://doi.org/10.23940/ijpe.20.10.p10.15881597
33.
MahajanK.LargeD. R.BurnettG.VelagaN. R. (2021a). Exploring the effectiveness of a digital voice assistant to maintain driver alertness in partially automated vehicles. Traffic Injury Prevention, 22(5), 378–383. https://doi.org/10.1080/15389588.2021.1904138
34.
MahajanK.LargeD. R.BurnettG.VelagaN. R. (2021b). Exploring the benefits of conversing with a digital voice assistant during automated driving: A parametric duration model of takeover time. Transportation Research Part F: Traffic Psychology and Behaviour, 80(1), 104–126. https://10.1016/j.trf.2021.03.012
35.
MartiP.JallaisC.KoustanaïA.GuillaumeA.MarsF. (2022). Impact of the driver's visual engagement on situation awareness and takeover quality. Transportation Research Part F: Traffic Psychology and Behaviour, 87(2), 391–402. https://doi.org/10.1016/j.trf.2022.04.018
36.
McWilliamsT.WardN. (2021). Underload on the road: Measuring vigilance decrements during partially automated driving. Frontiers in Psychology, 12(1), Article 631364. https://doi.org/10.3389/fpsyg.2021.631364
37.
MehlerB. (2020). Is supportive driver monitoring needed to maximize trust, use, and the safety-benefits of collaborative automation? In UNECE Global Forum for Road Traffic Safety Pre-event, Global Ministerial Conference on Road Safety Panel II: Emerging automotive technologies - how our life will change. UNECE.
38.
MihaljT.LiH.BabićD.LexC.JeudyM.ZovakG.BabićD.EichbergerA.EichbergerA. (2022). Road infrastructure challenges faced by automated driving: A review. Applied Sciences, 12(7), 3477. https://doi.org/10.3390/app12073477
39.
MillerD.SunA.JohnsM.IveH.SirkinD.AichS.JuW. (2015). Distraction becomes engagement in automated driving. In Proceedings of the Human Factors and Ergonomics Society - Annual Meeting (Vol. 59, pp. 1676–1680). Sage Publications. https://doi.org/10.1177/1541931215591362
40.
MishlerS.ChenJ. (2023). Boring but demanding: Using secondary tasks to counter the driver vigilance decrement for partially automated driving. Human Factors, 66(6), 1798–1811. https://10.1177/00187208231168697
41.
MüllerA. L.Fernandes-EstrelaN.HetfleischR.ZechaL.AbendrothB. (2021). Effects of non-driving related tasks on mental workload and take-over times during conditional automated driving. European Transport Research Review, 13(1), 16. https://doi.org/10.1186/s12544-021-00475-5
NaujoksF.HöflingS.PuruckerC.ZeebK. (2018). From partial and high automation to manual driving: Relationship between non-driving related tasks, drowsiness and take-over performance. Accident Analysis & Prevention, 121(1), 28–42. https://doi.org/10.1016/j.aap.2018.08.018
44.
NeubauerC.MatthewsG.SaxbyD. (2014). Fatigue in the automated vehicle: Do games and conversation distract or energize the driver? In Proceedings of the Human Factors and Ergonomics Society - Annual Meeting (Vol. 58, pp. 2053–2057). Sage Publications. https://doi.org/10.1177/1541931214581432
45.
PflegingB.RangM.BroyN. (2016). Investigating user needs for non-driving-related activities during automated driving. In Proceedings of the 15th international conference on mobile and ubiquitous multimedia (pp. 91–99). ACM.
46.
RadwinR.LeeJ. D.AkkasO. (2017). Driver movement patterns indicate distraction and engagement. Human Factors, 59(5), 844–860. https://doi.org/10.1177/0018720817696496
47.
SAE International. (2021). Taxonomy and definitions for terms related to driving automation systems for on-road motor vehicles (J3016_202104). SAE International. https://www.sae.org/standards/content/j3016_202104/
48.
SaitoY.WatahikiY.LeungC.ZhouH.ItohM. (2022). Effect of verbal messages with reminders to communicate driving situations to alter driver behavior in conditional driving automation. Transportation Research Part F: Traffic Psychology and Behaviour, 85(6), 69–82. https://doi.org/10.1016/j.trf.2022.01.003
49.
SallR.FengJ. (2019). Dual-target hazard perception: Could identifying one hazard hinder a driver's capacity to find a second?Accident Analysis & Prevention, 131(1), 213–224. https://doi.org/10.1016/j.aap.2019.06.016
SaryazdiR.BakK.CamposJ. L. (2019). Inattentional blindness during driving in younger and older adults. Frontiers in Psychology, 10, 880. https://doi.org/10.3389/fpsyg.2019.00880
52.
SaxbyD. J.MatthewsG.WarmJ. S.HitchcockE. M.NeubauerC. (2013). Active and passive fatigue in simulated driving: Discriminating styles of workload regulation and their safety impacts. Journal of Experimental Psychology: Applied, 19(4), 287–300. https://doi.org/10.1037/a0034386
53.
SchömigN.HarguttV.NeukumA.Petermann-StockI.OthersenI. (2015). The interaction between highly automated driving and the development of drowsiness. Procedia Manufacturing, 3, 6652–6659. https://doi.org/10.1016/j.promfg.2015.11.005
54.
SeamanS.GershonP.AngellL.MehlerB.ReimerB. (2022). Non-driving-related task engagement: The role of speed. Safety Now, 8(2), 34. https://doi.org/10.3390/safety8020034
55.
SteinbergerF.SchroeterR.WatlingC. N. (2017). From road distraction to safe driving: Evaluating the effects of boredom and gamification on driving behaviour, physiological arousal, and subjective experience. Computers in Human Behavior, 75, 714–726. https://doi.org/10.1016/j.chb.2017.06.019
56.
SümerÖ.GoldbergP.D’MelloS.GerjetsP.TrautweinU.KasneciE. (2021). Multimodal engagement analysis from facial videos in the classroom. IEEE Transactions on Affective Computing, 14(2), 1012–1027. https://doi.org/10.1109/taffc.2021.3127692
57.
TeohE. R. (2020). What’s in a name? Drivers’ perceptions of the use of five SAE level 2 driving automation systems. Journal of Safety Research, 72, 145–151. https://doi.org/10.1016/j.jsr.2019.11.005
58.
VictorT. W.TivestenE.GustavssonP.JohanssonJ.SangbergF.Ljung AustM. (2018). Automation expectation mismatch: Incorrect prediction despite eyes on threat and hands on wheel. Human Factors, 60(8), 1095–1116. https://doi.org/10.1177/0018720818788164
59.
WandtnerB.SchömigN.SchmidtG. (2018a). Effects of non-driving related task modalities on takeover performance in highly automated driving. Human Factors, 60(6), 870–881. https://doi.org/10.1177/0018720818768199
60.
WandtnerB.SchömigN.SchmidtG. (2018b). Secondary task engagement and disengagement in the context of highly automated driving. Transportation Research Part F: Traffic Psychology and Behaviour, 58, 253–263. https://doi.org/10.1016/j.trf.2018.06.001
61.
WeaverB. W.DeLuciaP. R. (2022). A systematic review and meta-analysis of takeover performance during conditionally automated driving. Human Factors, 64(7), 1227–1260. https://doi.org/10.1177/0018720820976476
62.
WickensC. D. (1984). Processing resources in attention. In ParasuramanR.DaviesD. R. (Eds.), Varieties of attention (pp. 63–102): Academic Press.
63.
WickensC. D. (2002). Multiple resources and performance prediction. Theoretical Issues in Ergonomics Science, 3(2), 159–177. https://doi.org/10.1080/14639220210123806
64.
WilsonJ.R.LorenzK.A. (2015). Generalized estimating equations logistic regression. In Modeling binary correlated responses using SAS, SPSS and R. ICSA book series in statistics (Vol. 9, pp. 103–130). Springer. https://doi.org/10.1007/978-3-319-23805-0_6
65.
YangS.KuoJ.LennéM. G. (2021). Effects of distraction in on-road level 2 automated driving: Impacts on glance behavior and takeover performance. Human Factors, 63(8), 1485–1497. https://doi.org/10.1177/0018720820936793
66.
YerkesR. M.DodsonJ. D. (1908). The relation of strength of stimulus to rapidity of habit-formation. Journal of Comparative Neurology and Psychology, 18(5), 459–482. https://doi.org/10.1002/cne.920180503
67.
YoungM. S.StantonN. A. (2002). Attention and automation: New perspectives on mental underload and performance. Theoretical Issues in Ergonomics Science, 3(2), 178–194. https://doi.org/10.1080/14639220210123789
