The Divergence of Objective and Subjective Situation Awareness: A Meta-Analysis

Abstract

A review of 37 studies that included both objective and subjective measures of situation awareness (SA) was conducted. Objective and subjective measures of SA were found to diverge across a wide range of measurement techniques. Reasons for these differences include a lack of meta-awareness about one’s own SA, poor SA/confidence calibration, and confounds with workload among some measures. A model that shows how objective and subjective SA combine to affect performance is presented.

Keywords

design evaluation measurement situation awareness

Get full access to this article

View all access options for this article.

References

Alexander

A. L.

Wickens

C. D.

(2004). Measuring traffic awareness in an integrated hazard display. In Proceedings of the Human Factors and Ergonomics Society Annual Meeting (pp. 171–175). Los Angeles, CA: SAGE.

Alexander

A. L.

Wickens

C. D.

(2005). Synthetic vision systems: Flightpath tracking, situation awareness, and visual scanning in an integrated hazard display. In Proceedings of the International Symposium on Aviation Psychology (pp. 1–6). Dayton, OH: Wright State University.

Bell

H. H.

Lyon

D. R.

(2000). Using observer ratings to assess situation awareness. In Endsley

M. R.

Garland

D. J.

(Eds.), Situation awareness analysis and measurement (pp. 129–146). Mahwah, NJ: Lawrence Erlbaum.

Cak

Say

Misirlisoy

(2019). Effects of working memory, attention, and expertise on pilots’ situation awareness. Cognition, Technology & Work. Advance online publication. doi:10.1007/s10111-019-00551-w/

Chiappe

D. L.

Rorie

R. C.

Moran

C. A.

K. P. L.

(2013). A situated approach to the acquisition of shared SA in team contexts. Theoretical Issues in Ergonomics Science, 15, 69–87.

Christ

R. E.

McKeever

K. J.

Huff

J. W.

(1994). Collective training of multiple team organizational units: Influence of intra- and inter-team processes. In Bradley

G. E.

Hendrick

H. W.

(Eds.), Human factors in organizational design and management—IV (pp. 323–326). Amsterdam, The Netherlands: Elsevier.

Cunningham

J. C.

Battiste

Curtis

Hallett

E. C.

Koltz

Brandt

S. L.

Johnson

W. W.

(2015). Measuring situation awareness with probe questions: Reasons for not answering the probes. Procedia Manufacturing, 3, 2982–2989.

Durso

F. T.

Bleckley

M. K.

Dattel

A. R.

(2006). Does situation awareness add to the validity of cognitive tests? Human Factors, 48, 721–733.

Durso

F. T.

Hackworth

C. A.

Truitt

T. R.

Crutchfield

Nikolic

Manning

C. A.

(1998). Situation awareness as a predictor of performance for en route air traffic controllers. Air Traffic Control Quarterly, 6(1), 1–20.

10.

Edgar

G. K.

Edgar

H. E.

Curry

M. B.

(2003). Using signal detection theory to measure situation awareness in command and control. In Proceedings of the Human Factors and Ergonomics Society 47th Annual Meeting (pp. 2019–2023). Santa Monica, CA: Sage.

11.

Einhorn

H. J.

Hogarth

R. M.

(1981). Behavioral decision theory: Processes of judgement and choice. Annual Review of Psychology, 32(1), 53–88.

12.

Endsley

M. R.

(1990a). A methodology for the objective measurement of situation awareness. In Situational awareness in aerospace operations (AGARD-CP-478) (pp. 1/1–1/9). Neuilly Sur Seine, France: North Atlantic Treaty Organization-Advisory Group for Aerospace Research and Development.

13.

Endsley

M. R.

(1990b). Predictive utility of an objective measure of situation awareness. In Proceedings of the Human Factors Society 34th Annual Meeting (pp. 41–45). Santa Monica, CA: Human Factors Society.

14.

Endsley

M. R.

(1993). A survey of situation awareness requirements in air-to-air combat fighters. International Journal of Aviation Psychology, 3, 157–168.

15.

Endsley

M. R.

(1995a). Measurement of situation awareness in dynamic systems. Human Factors, 37, 65–84.

16.

Endsley

M. R.

(1995b). Toward a theory of situation awareness in dynamic systems. Human Factors, 37, 32–64.

17.

Endsley

M. R.

(2000). Direct measurement of situation awareness: Validity and use of SAGAT. In Endsley

M. R.

Garland

D. J.

(Eds.), Situation awareness analysis and measurement (pp. 147–174). Mahwah, NJ: Lawrence Erlbaum.

18.

Endsley

M. R.

(2018a). Combating information attacks in the age of the internet: New challenges for cognitive engineering. Human Factors, 60, 1081–1094.

19.

Endsley

M. R.

(2018b). Expertise and situation awareness. In Ericsson

K. A.

Hoffman

R. R.

Kozbelt

Williams

A. M.

(Eds.), Cambridge handbook of expertise and expert performance (2nd ed., pp. 714–744). Cambridge, UK: Cambridge University Press.

20.

Endsley

M. R.

(in press). A systematic review and meta-analysis of direct, objective measures of SA: A comparison of SAGAT and SPAM. Human Factors.

21.

Endsley

M. R.

Bolstad

C. A.

(1994). Individual differences in pilot situation awareness. International Journal of Aviation Psychology, 4, 241–264.

22.

Endsley

M. R.

Garland

D. J.

(Eds.). (2000). Situation awareness analysis and measurement. Mahwah, NJ: Lawrence Erlbaum.

23.

Endsley

M. R.

Jones

D. G.

(2012). Designing for situation awareness: An approach to human-centered design (2nd ed.). London, England: Taylor & Francis.

24.

Endsley

M. R.

Jones

W. M.

(1997). Situation awareness, information dominance, and information warfare (AL/CF-TR-1997–0156). Wright-Patterson AFB, OH: United States Air Force Armstrong Laboratory.

25.

Endsley

M. R.

Selcon

S. J.

Hardiman

T. D.

Croft

D. G.

(1998). A comparative evaluation of SAGAT and SART for evaluations of situation awareness. In Proceedings of the Human Factors and Ergonomics Society Annual Meeting (pp. 82–86). Santa Monica, CA: Human Factors and Ergonomics Society.

26.

Endsley

M. R.

Sollenberger

Stein

(2000). Situation awareness: A comparison of measures. In Proceedings of the Human Performance, Situation Awareness and Automation: User-Centered Design for the New Millennium (pp. 15–19). Savannah, GA: SA Technologies.

27.

Fleming

S. M.

Lau

H. C.

(2014). How to measure metacognition. Frontiers in Human Neuroscience, 8, Article 443.

28.

Gatsoulis

Virk

G. S.

Dehghani-Sanij

A. A.

(2010). On the measurement of situation awareness for effective human-robot interaction in teleoperated systems. Journal of Cognitive Engineering and Decision Making, 4, 69–98.

29.

Gonzalez

Wimisberg

(2007). Situation awareness in dynamic decision making: Effects of practice and working memory. Journal of Cognitive Engineering and Decision Making, 1, 56–74.

30.

Grimaldi

Lau

Basso

M. A.

(2015). There are things that we know that we know, and there are things that we do not know we do not know: Confidence in decision-making. Neuroscience & Biobehavioral Reviews, 55, 88–97.

31.

Gutzwiller

R. S.

Clegg

B. A.

(2013). The role of working memory in levels of situation awareness. Journal of Cognitive Engineering and Decision Making, 7, 141–154.

32.

Hamilton

Mancuso

Mohammed

Tesler

McNeese

(2017). Skilled and unaware: The interactive effects of team cognition, team metacognition, and task confidence on team performance. Journal of Cognitive Engineering and Decision Making, 11, 382–395.

33.

Hauss

Eyferth

(2003). Securing future ATM-concepts’ safety by measuring situation awareness in ATC. Aerospace Science and Technology, 7, 417–427.

34.

Hogg

D. N.

Folles⊘

Strand-Volden

Torralba

(1995). Development of a situation awareness measure to evaluate advanced alarm systems in nuclear power plant control rooms. Ergonomics, 38, 2394–2413.

35.

Jipp

Ackerman

P. L.

(2016). The impact of higher levels of automation on performance and situation awareness: A function of information-processing ability and working-memory capacity. Journal of Cognitive Engineering and Decision Making, 10, 138–166.

36.

Jones

D. G.

Endsley

M. R.

(1996). Sources of situation awareness errors in aviation. Aviation, Space, and Environmental Medicine, 67, 507–512.

37.

Jones

D. G.

Endsley

M. R.

(2000a). Can real-time probes provide a valid measure of situation awareness? In Proceedings of the Human Performance, Situation Awareness and Automation: User-Centered Design for the New Millennium Conference (pp. 245–250). Savannah, GA: SA Technologies.

38.

Jones

D. G.

Endsley

M. R.

(2000b). Overcoming representational errors in complex environments. Human Factors, 42, 367–378.

39.

Jones

D. G.

Endsley

M. R.

(2004). Using real time probes for measuring situation awareness. International Journal of Aviation Psychology, 14, 343–367.

40.

Kruger

Dunning

(1999). Unskilled and unaware of it: How difficulties in recognizing one’s own incompetence lead to inflated self-assessments. Journal of Personality and Social Psychology, 77, 1121–1134.

41.

Lee

J. D.

(1999). Measuring driver adaptation to in-vehicle information systems: Disassociation of subjective and objective situation awareness measures. In Proceedings of the Human Factors and Ergonomics Society Annual Meeting (pp. 992–996). Los Angeles, CA: SAGE.

42.

Sanderson

Memisevic

Wong

(2007). Adapting situational awareness measures for hydropower display evaluations. In Proceedings of the Human Factors and Ergonomics Society Annual Meeting (pp. 210–214). Los Angeles, CA: SAGE.

43.

Lichacz

F. M.

(2008). Augmenting understanding of the relationship between situation awareness and confidence using calibration analysis. Ergonomics, 51, 1489–1502.

44.

Lichacz

F. M.

(2009). Calibrating situation awareness and confidence within a multinational coalition operation. Military Psychology, 21, 412–426.

45.

Lichacz

F. M.

Cain

Patel

(2003). Calibration of confidence in situation awareness queries. In Proceedings of the Human Factors and Ergonomics Society Annual Meeting (pp. 222–226). Los Angeles, CA: SAGE.

46.

J. C.

Sehic

Brookhuis

K. A.

Meijer

S. A.

(2016). Explicit or implicit situation awareness? Measuring the situation awareness of train traffic controllers. Transportation Research Part F: Traffic Psychology and Behaviour, 43, 325–338.

47.

Loft

Bowden

Braithwaite

Morrell

D. B.

Huf

Durso

F. T.

(2015). Situation awareness measures for simulated submarine track management. Human Factors, 57, 298–310.

48.

Loft

Morrell

D. B.

Huf

(2013). Using the situation present assessment method to measure situation awareness in simulated submarine track management. International Journal of Human Factors and Ergonomics, 2, 33–48.

49.

Loft

Morrell

D. B.

Ponton

Braithwaite

Bowden

Huf

(2016). The impact of uncertain contact location on situation awareness and performance in simulated submarine track management. Human Factors, 58, 1052–1068.

50.

Matthews

M. D.

Beal

S. A.

(2002). Assessing situation awareness in field training exercises. West Point, NY: U.S. Military Academy, Office of Military Psychology and Leadership.

51.

McGuinness

(2004). Quantitative analysis of situational awareness (QUASA): Applying signal detection theory to true/false probes and self-ratings. In Proceedings of the Command and Control Research and Technology Symposium (pp. 1–14). Bristol, UK: BAE Systems Advanced Technology Centre.

52.

McGuinness

Foy

J. L.

(2000). A subjective measure of situation awareness: The crew awareness rating scale. In Proceedings of the First Human Performance, Situation Awareness and Automation Conference (pp. 286–291). Savannah, GA: SA Technologies.

53.

Morgan

C. A.

Chiappe

D. L.

Kraut

Strybel

T. Z.

K. P. L.

(2012). An investigation into the potential sources of interference in SA probe techniques. In Landry

(Ed.), Advances in Human Aspects of Aviation (pp. 623–631). Boca Raton, FL: CRC Press.

54.

Naderpour

Zhang

(2016). A safety-critical decision support system evaluation using situation awareness and workload measures. Reliability Engineering & System Safety, 150, 147–159.

55.

Nickerson

R. S.

(1998). Confirmation bias: A ubiquitous phenomenon in many guises. Review of General Psychology, 2, 75–220.

56.

Nisbett

R. E.

Wilson

T. D.

(1977). Telling more than we can know: Verbal reports on mental processes. Psychological Review, 84, 231–259.

57.

Pierce

C. S.

Jastrow

(1885). On small differences in sensation. Memoirs of the National Academy of Sciences, 3, 73–83.

58.

Pierce

R. S.

(2012). The effect of SPAM administration during a dynamic simulation. Human Factors, 54, 838–848.

59.

Pierce

R. S.

Strybel

T. Z.

K.-P. L.

(2008). Comparing situation awareness measurement techniques in a low fidelity air traffic control simulation. In Proceedings of the 26th International Congress of the Aeronautical Sciences (ICAS) (pp. 1–8). Anchorage, AS: International Congress of the Aeronautical Sciences.

60.

Price

Tenan

Head

Maslin

LaFiandra

(2016). Acute stress causes over confidence in situation awareness. In Proceedings of the 2016 IEEE International Multi-Disciplinary Conference on Cognitive Methods in Situation Awareness and Decision Support (CogSIMA) (pp. 1–6). New York, NY: IEEE.

61.

Puuska

Rummukainen

Timonen

Lääperi

Klemetti

Oksama

Vankka

(2018). Nationwide critical infrastructure monitoring using a common operating picture framework. International Journal of Critical Infrastructure Protection, 20, 28–47.

62.

Rose

Bearman

Dorrian

(2018). The Low-Event Task Subjective Situation Awareness (LETSSA) technique: Development and evaluation of a new subjective measure of situation awareness. Applied Ergonomics, 68, 273–282.

63.

Rousseau

Tremblay

Banbury

Breton

Guitouni

(2010). The role of metacognition in the relationship between objective and subjective measures of situation awareness. Theoretical Issues in Ergonomics Science, 11, 119–130.

64.

Sætrevik

(2012). A controlled field study of situation awareness measures and heart rate variability in emergency handling teams. In Proceedings of the Human Factors and Ergonomics Society Annual Meeting (pp. 2006–2010). Los Angeles, CA: SAGE.

65.

Salmon

P. M.

Stanton

N. A.

Walker

G. H.

Jenkins

Ladva

Rafferty

Young

(2009). Measuring situation awareness in complex systems: Comparison of measures study. International Journal of Industrial Ergonomics, 39, 490–500.

66.

Salmon

P. M.

Stanton

N. A.

Young

K. L.

(2011). Situation awareness on the road: Review, theoretical and methodological issues and future directions. Theoretical Issues in Ergonomics Science, 13, 472–492.

67.

Sarter

N. B.

Woods

D. D.

(1991). Situation awareness: A critical but ill-defined phenomenon. International Journal of Aviation Psychology, 1, 45–57.

68.

Schuster

Keebler

J. R.

Jentsch

Zuniga

(2012). Comparison of SA measurement techniques in a human-robot team task. In Proceedings of the Human Factors and Ergonomics Society Annual Meeting (pp. 1713–1717). Los Angeles, CA: SAGE.

69.

Selcon

S. J.

Taylor

R. M.

(1990). Evaluation of the situational awareness rating technique (SART) as a tool for aircrew systems design. In Situational awareness in aerospace operations (AGARD-CP-478) (pp. 5/1–5/8). Neuilly Sur Seine, France: North Atlantic Treaty Organization-Advisory Group for Aerospace Research and Development.

70.

Selcon

S. J.

Taylor

R. M.

Koritsas

(1991). Workload or situational awareness? TLX vs. SART for aerospace systems design evaluation. In Proceedings of the Human Factors Society 35th Annual Meeting (pp. 62–66). Santa Monica, CA: Human Factors Society.

71.

Selkowitz

A. R.

Lakhmani

S. G.

Chen

J. Y. C.

(2017). Using agent transparency to support situation awareness of the autonomous squad member. Cognitive Systems Research, 46, 13–25.

72.

Sethumadhavan

(2011). Knowing what you know: The role of meta-situation awareness in predicting situation awareness. In Proceedings of the Human Factors and Ergonomics Society Annual Meeting (pp. 360–364). Los Angeles, CA: SAGE.

73.

Shipstead

Lindsey

D. R.

Marshall

R. L.

Engle

R. W.

(2014). The mechanisms of working memory capacity: Primary memory, secondary memory, and attention control. Journal of Memory and Language, 72, 116–141.

74.

Sorensen

L. J.

Stanton

N. A.

(2011). Is SA shared or distributed in team work? An exploratory study in an intelligence analysis task. International Journal of Industrial Ergonomics, 41, 677–687.

75.

Strater

L. D.

Endsley

M. R.

Pleban

R. J.

Matthews

M. D.

(2001). Measures of platoon leader situation awareness in virtual decision making exercises (Research Report No. 1770). Alexandria, VA: Army Research Institute.

76.

Strybel

T. Z.

K.-P. L.

Dwyer

J. P.

Kraft

Ngo

T. K.

Chambers

Garcia

F. P.

(2007). Predicting perceived situation awareness of low altitude aircraft in terminal airspace using probe questions. In Proceedings of the International Conference on Human-Computer Interaction (pp. 939–948). New York, NY: Springer.

77.

Strybel

T. Z.

K.-P. L.

Kraft

Minakata

(2008). Assessing the situation awareness of pilots engaged in self spacing. In Proceedings of the Human Factors and Ergonomics Society Annual Meeting (pp. 11–15). Los Angeles, CA: SAGE.

78.

Sulistyawati

Wickens

C. D.

Chui

Y. P.

(2009). Exploring the concept of team situation awareness in a simulated air combat environment. Journal of Cognitive Engineering and Decision Making, 3, 309–330.

79.

Sulistyawati

Wickens

C. D.

Chui

Y. P.

(2011). Prediction in situation awareness: Confidence bias and underlying cognitive abilities. International Journal of Aviation Psychology, 21, 153–174.

80.

Taber

M. J.

McCabe

Klein

R. M.

Pelot

R. P.

(2013). Development and evaluation of an offshore oil and gas Emergency Response Focus Board. International Journal of Industrial Ergonomics, 43, 40–51.

81.

Trapsilawati

Wickens

C. D.

Chen

C.-H.

(2016). Benefits of imperfect conflict resolution advisory aids for future air traffic control. Human Factors, 58, 1007–1019.

82.

Van den Beukel

A. P.

van der Voort

M. C

. (2017). How to assess driver’s interaction with partially automated driving systems—A framework for early concept assessment. Applied Ergonomics, 59, 302–312.

83.

Vidulich

M. A.

(1989). The use of judgment matrices in subjective workload assessment: The subjective workload dominance (SWORD) technique. In Proceedings of the Human Factors Society 33rd Annual Meeting (pp. 1406–1410). Santa Monica, CA: Human Factors Society.

84.

Walker

G. H.

Stanton

N. A.

Young

M. S.

(2006). The ironies of vehicle feedback in car design. Ergonomics, 49, 161–179.

85.

Walker

G. H.

Stanton

N. A.

Young

M. S.

(2008). Feedback and driver situation awareness (SA): A comparison of SA measures and contexts. Transportation Research Part F: Traffic Psychology and Behaviour, 11, 282–299.

86.

Wickens

C. D.

(2008). Multiple resources and mental workload. Human Factors, 50, 449–455.

87.

Willems

B. F.

Heiney

(2001). Real-time assessment of situation awareness of air traffic control specialists on operational host computer system and display system replacement hardware. In Proceedings of the USA/Europe Air Traffic Management R&D Seminar (pp. 1–11). Santa Fe, NM: Federal Aviation Administration/Eurocontrol.

88.

Yerkes

R. M.

Dodson

J. D.

(1908). The relation of strength of stimulus to rapidity of habit-formation. The Journal of Comparative Neurology, 18, 459–482.

89.

Yeung

Summerfield

(2012). Metacognition in human decision-making: Confidence and error monitoring. Philosophical Transactions of the Royal Society B: Biological Sciences, 367, 1310–1321.