The rapid growth of autonomous vehicles is expected to improve roadway safety. However, certain levels of vehicle automation will still require drivers to ‘takeover’ during abnormal situations, which may lead to breakdowns in driver-vehicle interactions. To date, there is no agreement on how to best support drivers in accomplishing a takeover task. Therefore, the goal of this study was to investigate the effectiveness of multimodal alerts as a feasible approach. In particular, we examined the effects of uni-, bi-, and trimodal combinations of visual, auditory, and tactile cues on response times to takeover alerts. Sixteen participants were asked to detect 7 multimodal signals (i.e., visual, auditory, tactile, visual-auditory, visual-tactile, auditory-tactile, and visual-auditory-tactile) while driving under two conditions: with SAE Level 3 automation only or with SAE Level 3 automation in addition to performing a road sign detection task. Performance on the signal and road sign detection tasks, pupil size, and perceived workload were measured. Findings indicate that trimodal combinations result in the shortest response time. Also, response times were longer and perceived workload was higher when participants were engaged in a secondary task. Findings may contribute to the development of theory regarding the design of takeover request alert systems within (semi) autonomous vehicles.
Get full access to this article
View all access options for this article.
References
1.
BrickmanB. J.HettingerL. J.HaasM. W. (2000). Multisensory interface design for complex task domains: Replacing information overload with meaning in tactical crew stations. International Journal of Aviation Psychology, 10(3), 273–290.
2.
ForsterB.Cavina-PratesiC.AgliotiS. M.BerlucchiG. (2002). Redundant target effect and intersensory facilitation from visual-tactile interactions in simple reaction time. Experimental Brain Research, 143(4), 480–487.
3.
GhuntlaT.GokhaleP.MehtaH.ShahC. (2014). A comparison and importance of auditory and visual reaction time in basketball players. Saudi Journal of Sports Medicine, 14(1), 35.
4.
HartS. G.StavelandL. E. (1988). Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research. Advances in Psychology, 52, 139–183.
5.
HechtD.ReinerM.KarniA. (2008). Multisensory enhancement: gains in choice and in simple response times. Experimental Brain Research, 189(2), 133–143.
6.
HoC.-Y.NikolicM. I.SarterN. B. (2001). Supporting Timesharing and Interruption Management Through Multimodal Information Presentation. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 45(4), 341–345.
7.
JainA.BansalR.KumarA.SinghK. D. (2015). A comparative study of visual and auditory reaction times on the basis of gender and physical activity levels of medical first year students. International Journal of Applied & Basic Medical Research, 5(2), 124–127.
KörberM.PraschL.BenglerK. (2018). Why Do I Have to Drive Now? Post Hoc Explanations of Takeover Requests. Human Factors, 60(3), 305–323.
10.
KramerA. F.CassavaughN.HorreyW. J.BecicE.MayhughJ. L. (2007). Influence of Age and Proximity Warning Devices on Collision Avoidance in Simulated Driving. Human Factors: The Journal of the Human Factors and Ergonomics Society, 49(5), 935–949.
11.
LatorellaK. A. (1999). Investigating Interruptions: Implications for Flightdeck Performance. Hampton, VA.
12.
LiS.BlytheP.GuoW.NamdeoA. (2018). Investigation of older driver’s takeover control performance in highly automated vehicles in adverse weather conditions. IET Intelligent Transport Systems, 12(9), 1157-1165.
13.
LitmanT. (2018). Autonomous Vehicle Implementation Predictions Implications for Transport Planning (No. 15-3326).
14.
LlanerasR. E.SalingerJ.GreenC. A. (2013). Human factors issues associated with limited ability autonomous driving systems: Drivers’ allocation of visual attention to the forward roadway. Proceedings of the Seventh International Driving Symposium on Human Factors in Driver Assessment, Training, and Vehicle Design, 92–98.
15.
LundqvistL. M.ErikssonL. (2019). Age, cognitive load, and multimodal effects on driver response to directional warning. Applied Ergonomics, 76(January), 147–154.
16.
MarquartG.CabrallC.de WinterJ. (2015). Review of Eye-related Measures of Drivers’ Mental Workload. Procedia Manufacturing, 3, 2854–2861.
17.
MolnarL. J. (2017). Age-Related Differences in Driver Behavior Associated with Automated Vehicles and the Transfer of Control between Automated and Manual Control: A Simulator Evaluation. University of Michigan.
18.
National Center for Statistics and Analysis. (2018). 2017 fatal motor vehicle crashes: Overview.
19.
National Highway Traffic Safety Administration. (2017). Automated Driving Systems 2.0: A Vision for Safety. U.S. Department of Transportation.
20.
PetermeijerS.BazilinskyyP.BenglerK.de WinterJ. (2017). Take-over again: Investigating multimodal and directional TORs to get the driver back into the loop. Applied Ergonomics, 62, 204–215.
21.
PittsB. J.SarterN. (2018). What You Don’t Notice Can Harm You: Age-Related Differences in Detecting Concurrent Visual, Auditory, and Tactile Cues. Human Factors, 60(4), 445–464.
22.
PolitisI.BrewsterS.Frank Pollickglasgowacuk. (2013). Evaluating Multimodal Driver Displays of Varying Urgency. In Proceedings of the 5th International Conference on Automotive User Interfaces and Interactive Vehicular Applications (pp. 92-99). ACM.
23.
PolitisI.BrewsterS.PollickF. (2014). Evaluating multimodal driver displays under varying situational urgency. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, (ACM), 4067–4076.
24.
PolitisI.BrewsterS.PollickF. (2015). Language-based multimodal displays for the handover of control in autonomous cars. In Proceedings of the 7th International Conference on Automotive User Interfaces and Interactive Vehicular Applications - AutomotiveUI ’15 (pp. 3–10).
25.
PolitisI.BrewsterS.PollickF. (2017). Using Multimodal Displays to Signify Critical Handovers of Control to Distracted Autonomous Car Drivers. International Journal of Mobile Human Computer Interaction, 9(3), 1–16.
26.
PomplunM.SunkaraS. (2013). Pupil Dilation as an Indicator of Cognitive Workload in Human-Computer Interaction. Proceedings of the International Conference on HCI, (May), 542–546.
27.
SarterN. B. (2006). Multimodal information presentation: Design guidance and research challenges. International Journal of Industrial Ergonomics, 36, 439–445.
28.
SatterfieldK.RamirezR.ShawT.ParasuramanR. (2012). Measuring workload during a dynamic supervisory control task using cerebral blood flow velocity and the NASA-TLX. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 56(1), 163–167.
29.
SheltonJ.KumarG. P. (2010). Comparison between Auditory and Visual Simple Reaction Times. Neuroscience and Medicine, 01(01), 30–32.
30.
SuiedC.SusiniP.McAdamsS. (2008). Evaluating warning sound urgency with reaction times. Journal of Experimental Psychology: Applied, 14(3), 201–212.
31.
WickensC. D. (2008). Multiple Resources and Mental Workload. Human Factors: The Journal of the Human Factors and Ergonomics Society, 50(3), 449–455.
32.
YoonS. H.KimY. W.JiY. G. (2019). The effects of takeover request modalities on highly automated car control transitions. Accident Analysis and Prevention, 123(November2018), 150–158.
