The Effects of Auditory Working Memory Task on Situation Awareness in Complex Dynamic Environments: An Eye-movement Study

Abstract

Objective

This study investigated the effect of auditory working memory task on situation awareness (SA) and eye-movement patterns in complex dynamic environments.

Background

Many human errors in aviation are caused by a lack of SA, and distraction from auditory secondary tasks is a serious threat to SA. However, it remains unclear how auditory working memory tasks affect SA and eye-movement patterns.

Method

Participants (n = 28) were randomly allocated to two groups and received different periods of visual search training (short versus long). They subsequently completed a situation awareness measurement task in three auditory secondary task conditions (without secondary task, auditory calculation task, and auditory 2-back task). Eye-movement data were collected during the situation awareness measurement task.

Results

The auditory 2-back task significantly reduced overall SA, Level 1 SA, dwell times, and total percentage of fixation time on task-related areas of interests in the SA measurement task. Overall SA and Level 3 SA were not reduced by the auditory 2-back task in individuals in the longer visual search training time condition.

Conclusion

Auditory working memory load impairs SA in the perception and projection stage; however, greater experience can overcome impairment of SA in the projection stage.

Application

This study provided possible approaches to preventing loss of SA: (1) improving crew members’ communication skills to ensure the accurate and clear transmission of information, reducing the difficulty of processing information, and (2) providing targeted cognitive training tailored to each pilot’s level of experience.

Keywords

situational awareness secondary task working memory eye-tracking technique

Get full access to this article

View all access options for this article.

References

Argyle

E. M.

Gourley

J. J.

Kang

Shehab

R. L.

(2020). Investigating the relationship between eye movements and situation awareness in weather forecasting. Applied Ergonomics, 85, 103071. https://doi.org/10.1016/j.apergo.2020.103071

ATSB . (2006). Dangerous distraction: An examination of accidents and incidents involving pilot distraction in Australia between 1997 and 2004. www.atsb.gov.au/publications/2005/distraction_report/

Bolstad

C. A.

Endsley

M. R.

Costello

A. M.

Howell

C. D.

(2010). Evaluation of computer-based situation awareness training for general aviation pilots. International Journal of Aviation Psychology, 20(3), 269–294. https://doi.org/10.1080/10508414.2010.487013

Cak

Say

Misirlisoy

(2020). Effects of working memory, attention, and expertise on pilots’ situation awareness. Cognition, Technology and Work, 22(1), 85–94. https://doi.org/10.1007/s10111-019-00551-w

Castro

Ventsislavova

Peña-Suarez

Gugliotta

Garcia-Fernandez

Eisman

Crundall

(2016). Proactive listening to a training commentary improves hazard prediction. Safety science, 82, 144–154. https://doi.org/10.1016/j.ssci.2015.09.018

Chaparro

Tokuda

Morris

(2007). Effects of visual-spatial and verbal working memory load on visual attention and driving performance. Journal of Vision, 7(9), 683–683. https://doi.org/10.1167/7.9.683

Collette

Van der Linden

(2002). Brain imaging of the central executive component of working memory. Neuroscience & Biobehavioral Reviews, 26(2), 105–125. https://doi.org/10.1016/S0149-7634(01)00063-X

Dahlman

Sjörs

Lindström

Ledin

Falkmer

(2009). Performance and Autonomic Responses During Motion Sickness. Human Factors, 51(1), 56–66. https://doi.org/10.1177/0018720809332848

Dehais

Roy

R. N.

Scannella

(2019). Inattentional deafness to auditory alarms: Inter-individual differences, electrophysiological signature and single trial classification. Behavioural Brain Research, 360, 51–59. https://doi.org/10.1016/j.bbr.2018.11.045

10.

Doane

S. M.

Sohn

Y. W.

Jodlowski

M. T.

(2004). Pilot ability to anticipate the consequences of flight actions as a function of expertise. Human Factors, 46(1), 92–103. https://doi.org/10.1518/hfes.46.1.92.30386

11.

Ebadi

Fisher

D. L.

Roberts

S. C.

(2019). Impact of cognitive distractions on drivers’ hazard anticipation behavior in complex scenarios. Transportation research record, 2673(9), 440–451. https://doi.org/10.1177/0361198119846463

12.

Ebadi

Pai

Samuel

Fisher

D. L.

(2020). Impact of cognitive distractions on drivers’ hazardous event anticipation and mitigation behavior in vehicle–bicycle conflict situations. Transportation research record, 2674(7), 504–513. https://doi.org/10.1177/0361198120923660

13.

Endsley

M. R.

(1988). Design and evaluation for situation awareness enhancement. In Proceedings of the Human Factors Society annual meeting (Vol. 32, No. 2, pp. 97–101). Sage CA: Los Angeles, CA: Sage Publications.

14.

Endsley

M. R.

(1995a). Toward a theory of situation awareness in dynamic systems. Human Factors, 37(1), 32–64. https://doi.org/10.1518/001872095779049543

15.

Endsley

M. R.

(1995b). Measurement of situation awareness in dynamic systems. Human Factors, 37(1), 65–84. https://doi.org/10.1518/001872095779049499

16.

Endsley

M. R.

(2000). Errors in situation assessment: Implications for system design. In Human error and system design and management (pp. 15–26). Springer. https://doi.org/10.1007/BFb0110451

17.

Endsley

M. R.

(2015). Situation awareness misconceptions and misunderstandings. Journal of Cognitive Engineering and Decision Making, 9(1), 4–32. https://doi.org/10.1177/1555343415572631

18.

Endsley

M. R.

(2021). A systematic review and meta-analysis of direct objective measures of situation awareness: a comparison of SAGAT and SPAM. Human factors, 63(1), 124–150. https://doi.org/10.1177/0018720819875376

19.

Endsley

M. R.

Bolstad

C. A.

(1994). Individual differences in pilot situation awareness. International Journal of Aviation Psychology, 4(3), 241–264. https://doi.org/10.1207/s15327108ijap0403_3

20.

Engström

Aust

M. L.

Viström

(2010). Effects of working memory load and repeated scenario exposure on emergency braking performance. Human factors, 52(5), 551–559. https://doi.org/10.1177/0018720810381072

21.

Faul

Erdfelder

Lang

A. G.

Buchner

(2007). G* Power 3: A flexible statistical power analysis program for the social, behavioral, and biomedical sciences. Behavior research methods, 39(2), 175–191. https://doi.org/10.3758/bf03193146

22.

Gonzalez

Wimisberg

(2007). Situation awareness in dynamic decision making: Effects of practice and working memory. Journal of Cognitive Engineering and Decision Making, 1(1), 56–74. https://doi.org/10.1177/155534340700100103

23.

Grundgeiger

Hohm

Michalek

Egenolf

Markus

Happel

(2022). The validity of the SEEV Model as a process measure of situation awareness: the example of a simulated endotracheal intubation. Human Factors, 64(7), 1181–1194. https://doi.org/10.1177/0018720821991651

24.

Harbluk

J. L.

Noy

Y. I.

Trbovich

P. L.

Eizenman

(2007). An on-road assessment of cognitive distraction: Impacts on drivers’ visual behavior and braking performance. Accident Analysis and Prevention, 39(2), 372–379. https://doi.org/10.1016/j.aap.2006.08.013

25.

Heenan

Herdman

C. M.

Brown

M. S.

Robert

(2014). Effects of conversation on situation awareness and working memory in simulated driving. Human Factors, 56(6), 1077–1092. https://doi.org/10.1177/0018720813519265

26.

Hollands

J. G.

Spivak

Kramkowski

E. W.

(2019). Cognitive load and situation awareness for soldiers: effects of message presentation rate and sensory modality. Human factors, 61(5), 763–773. https://doi.org/10.1177/0018720819825803

27.

Johannsdottir

K. R.

Herdman

C. M.

(2010). The role of working memory in supporting drivers’ situation awareness for surrounding traffic. Human Factors, 52(6), 663–673. https://doi.org/10.1177/0018720810385427

28.

Jones

D. G.

Endsley

M. R.

(1996). Sources of situation awareness errors in aviation. Aviation, space, and environmental medicine, 67(6), 507–512.

29.

Kaber

Jin

Zahabi

Pankok

(2016). The effect of driver cognitive abilities and distractions on situation awareness and performance under hazard conditions. Transportation Research Part F: Traffic Psychology and Behaviour, 42, 177–194. https://doi.org/10.1016/j.trf.2016.07.014

30.

Kane

M. J.

Engle

R. W.

(2002). The role of prefrontal cortex in working-memory capacity, executive attention, and general fluid intelligence: An individual-differences perspective. Psychonomic bulletin & review, 9(4), 637–671. https://doi.org/10.3758/bf03196323

31.

Kass

S. J.

Cole

K. S.

Stanny

C. J.

(2007). Effects of distraction and experience on situation awareness and simulated driving. Transportation Research Part F: Traffic Psychology and Behaviour, 10(4), 321–329. https://doi.org/10.1016/j.trf.2006.12.002

32.

Kaur

Chaujar

Chinnadurai

(2020). Effects of neural mechanisms of pretask resting eeg alpha information on situational awareness: a functional connectivity approach. Human Factors, 62(7), 1150–1170. https://doi.org/10.1177/0018720819869129

33.

Kelly

Efthymiou

(2019). An analysis of human factors in fifty controlled flight into terrain aviation accidents from 2007 to 2017. Journal of safety research, 69, 155–165. https://doi.org/10.1016/j.jsr.2019.03.009

34.

Kharoufah

Murray

Baxter

Wild

(2018). A review of human factors causations in commercial air transport accidents and incidents: From to 2000–2016. Progress in Aerospace Sciences, 99, 1–13. https://doi.org/10.1016/j.paerosci.2018.03.002

35.

Kim

Gabbard

J. L.

(2022). Assessing distraction potential of augmented reality head-up displays for vehicle drivers. Human Factors, 64(5), 852–865. https://doi.org/10.1177/0018720819844845

36.

Lewkowicz

Strózak

Bałaj

Francuz

(2019). Auditory verbal working memory load effects on a simulator-induced spatial disorientation event. Aerospace Medicine and Human Performance, 90(6), 531–539. https://doi.org/10.3357/AMHP.5277.2019

37.

Kaber

D. B.

(2005). Situation awareness and workload in driving while using adaptive cruise control and a cell phone. International Journal of Industrial Ergonomics, 35(10), 939–953. https://doi.org/10.1016/j.ergon.2005.04.002

38.

Macdonald

J. S. P.

Lavie

(2011). Visual perceptual load induces inattentional deafness. Attention, Perception, and Psychophysics, 73(6), 1780–1789. https://doi.org/10.3758/s13414-011-0144-4

39.

Mansikka

Harris

Virtanen

(2019). Pilot competencies as components of a dynamic human‐machine system. Human Factors and Ergonomics in Manufacturing & Service Industries, 29(6), 466–477. https://doi.org/10.1002/hfm.20809

40.

Molesworth

B. R.

Estival

(2015). Miscommunication in general aviation: The influence of external factors on communication errors. Safety science, 73, 73–79. https://doi.org/10.1016/j.ssci.2014.11.004

41.

Moore

Gugerty

(2010). Development of a novel measure of situation awareness: The case for eye movement analysis. In Proceedings of the human factors and ergonomics society annual meeting (Vol. 54, No. 19, pp. 1650–1654). Sage CA: Los Angeles, CA: Sage Publications.

42.

Murphy

Greene

C. M.

(2015). High perceptual load causes inattentional blindness and deafness in drivers. Visual Cognition, 23(7), 810–814. https://doi.org/10.1080/13506285.2015.1093245

43.

Murphy

Greene

C. M.

(2016). Perceptual load induces inattentional blindness in drivers. Applied Cognitive Psychology, 30(3), 479–483. https://doi.org/10.1002/acp.3216

44.

Murphy

Greene

C. M.

(2017). The elephant in the road: Auditory perceptual load affects driver perception and awareness. Applied Cognitive Psychology, 31(2), 258–263. https://doi.org/10.1002/acp.3311

45.

Nguyen

Lim

C. P.

Nguyen

N. D.

Gordon-Brown

Nahavandi

(2019). A review of situation awareness assessment approaches in aviation environments. IEEE Systems Journal, 13(3), 3590–3603. https://doi.org/10.1109/JSYST.2019.2918283

46.

O'Brien

K. S.

O'Hare

(2007). Situational awareness ability and cognitive skills training in a complex real-world task. Ergonomics, 50(7), 1064–1091. https://doi.org/10.1080/00140130701276640

47.

Pammer

McKerral

Liu

(2021). The contribution of memory to differences in situation awareness in expert and non-expert drivers. Transportation Research Part F: Traffic Psychology and Behaviour, 82, 154–166. https://doi.org/10.1016/j.trf.2021.08.007

48.

Peißl

Wickens

C. D.

Baruah

(2018). Eye-tracking measures in aviation: A selective literature review. The International Journal of Aerospace Psychology, 28(3–4), 98–112. https://doi.org/10.1080/24721840.2018.1514978

49.

Raveh

Lavie

(2015). Load-induced inattentional deafness. Attention, Perception, and Psychophysics, 77(2), 483–492. https://doi.org/10.3758/s13414-014-0776-2

50.

Rogers

Zhang

Kaber

Liang

Gangakhedkar

(2011). The effects of visual and cognitive distraction on driver situation awareness. In International Conference on Engineering Psychology and Cognitive Ergonomics (pp. 186–195). Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-21741-8_21

51.

Salmela

V. R.

Moisala

Alho

(2014). Working memory resources are shared across sensory modalities. Attention, Perception, and Psychophysics, 76(7), 1962–1974. https://doi.org/10.3758/s13414-014-0714-3

52.

Sohn

Y. W.

Doane

S. M.

(2003). Roles of working memory capacity and long-term working memory skill in complex task performance. Memory and Cognition, 31(3), 458–466. https://doi.org/10.3758/BF03194403

53.

Sohn

Y. W.

Doane

S. M.

(2004). Memory processes of flight situation awareness: Interactive roles of working memory capacity, long-term working memory, and expertise. Human Factors, 46(3), 461–475. https://doi.org/10.1518/hfes.46.3.461.50392

54.

Stapel

El Hassnaoui

Happee

(2022). Measuring driver perception: combining eye-tracking and automated road scene perception. Human Factors, 64(4), 714–731. https://doi.org/10.1177/0018720820959958

55.

Strayer

D. L.

Fisher

D. L.

(2016). Spider: a framework for understanding driver distraction. Human factors, 58(1), 5–12. https://doi.org/10.1177/0018720815619074

56.

Sulistyawati

Wickens

C. D.

Chui

Y. P.

(2011). Prediction in situation awareness: Confidence bias and underlying cognitive abilities. The International Journal of Aviation Psychology, 21(2), 153–174. https://doi.org/10.1080/10508414.2011.556492

57.

Van Dam

Kass

S. J.

VanWormer

(2020). The effects of passive mobile phone interaction on situation awareness and driving performance. Journal of Transportation Safety and Security, 12(8), 1007–1024. https://doi.org/10.1080/19439962.2018.1564947

58.

van de Merwe

van Dijk

Zon

(2012). Eye movements as an indicator of situation awareness in a flight simulator experiment. International Journal of Aviation Psychology, 22(1), 78–95. https://doi.org/10.1080/10508414.2012.635129

59.

White

O’Hare

(2022). In plane sight: Inattentional blindness affects visual detection of external targets in simulated flight. Applied Ergonomics, 98, 103578. https://doi.org/10.1016/j.apergo.2021.103578

60.

Wood

Hartley

Furley

P. A.

Wilson

M. R.

(2016). Working memory capacity, visual attention and hazard perception in driving. Journal of Applied Research in Memory & Cognition, 5(4), 454–462. https://doi.org/10.1016/j.jarmac.2016.04.009

61.

Wickens

C. D.

(2008). Situation awareness: Review of Mica Endsley’s 1995 articles on situation awareness theory and measurement. Human Factors, 50(3), 397–403. https://doi.org/10.1518/001872008X288420

62.

Molesworth

B. R.

Estival

(2019). An investigation into the factors that affect miscommunication between pilots and air traffic controllers in commercial aviation. The international journal of aerospace psychology, 29(1–2), 53–63. https://doi.org/10.1080/24721840.2019.1604138

63.

Yang

Chen

Easa

S. M.

Zheng

(2020). Structural equation modeling of drivers’ situation awareness considering road and driver factors. Frontiers in Psychology, 11, 1601. https://doi.org/10.3389/fpsyg.2020.01601

64.

Zhang

Yang

Liang

Pitts

B. J.

Prakah-Asante

K. O.

Curry

Duerstock

B. S.

Wachs

J. P.

(2020). Physiological measurements of situation awareness: A systematic review. Human Factors, 0(0). https://doi.org/10.1177/0018720820969071