A review of 37 studies that included both objective and subjective measures of situation awareness (SA) was conducted. Objective and subjective measures of SA were found to diverge across a wide range of measurement techniques. Reasons for these differences include a lack of meta-awareness about one’s own SA, poor SA/confidence calibration, and confounds with workload among some measures. A model that shows how objective and subjective SA combine to affect performance is presented.
AlexanderA. L.WickensC. D. (2004). Measuring traffic awareness in an integrated hazard display. In Proceedings of the Human Factors and Ergonomics Society Annual Meeting (pp. 171–175). Los Angeles, CA: SAGE.
2.
AlexanderA. L.WickensC. D. (2005). Synthetic vision systems: Flightpath tracking, situation awareness, and visual scanning in an integrated hazard display. In Proceedings of the International Symposium on Aviation Psychology (pp. 1–6). Dayton, OH: Wright State University.
3.
BellH. H.LyonD. R. (2000). Using observer ratings to assess situation awareness. In EndsleyM. R.GarlandD. J. (Eds.), Situation awareness analysis and measurement (pp. 129–146). Mahwah, NJ: Lawrence Erlbaum.
4.
CakS.SayB.MisirlisoyM. (2019). Effects of working memory, attention, and expertise on pilots’ situation awareness. Cognition, Technology & Work. Advance online publication. doi:10.1007/s10111-019-00551-w/
5.
ChiappeD. L.RorieR. C.MoranC. A.VuK. P. L. (2013). A situated approach to the acquisition of shared SA in team contexts. Theoretical Issues in Ergonomics Science, 15, 69–87.
6.
ChristR. E.McKeeverK. J.HuffJ. W. (1994). Collective training of multiple team organizational units: Influence of intra- and inter-team processes. In BradleyG. E.HendrickH. W. (Eds.), Human factors in organizational design and management—IV (pp. 323–326). Amsterdam, The Netherlands: Elsevier.
7.
CunninghamJ. C.BattisteH.CurtisS.HallettE. C.KoltzM.BrandtS. L.JohnsonW. W. (2015). Measuring situation awareness with probe questions: Reasons for not answering the probes. Procedia Manufacturing, 3, 2982–2989.
8.
DursoF. T.BleckleyM. K.DattelA. R. (2006). Does situation awareness add to the validity of cognitive tests?Human Factors, 48, 721–733.
9.
DursoF. T.HackworthC. A.TruittT. R.CrutchfieldJ.NikolicD.ManningC. A. (1998). Situation awareness as a predictor of performance for en route air traffic controllers. Air Traffic Control Quarterly, 6(1), 1–20.
10.
EdgarG. K.EdgarH. E.CurryM. B. (2003). Using signal detection theory to measure situation awareness in command and control. In Proceedings of the Human Factors and Ergonomics Society 47th Annual Meeting (pp. 2019–2023). Santa Monica, CA: Sage.
11.
EinhornH. J.HogarthR. M. (1981). Behavioral decision theory: Processes of judgement and choice. Annual Review of Psychology, 32(1), 53–88.
12.
EndsleyM. R. (1990a). A methodology for the objective measurement of situation awareness. In Situational awareness in aerospace operations (AGARD-CP-478) (pp. 1/1–1/9). Neuilly Sur Seine, France: North Atlantic Treaty Organization-Advisory Group for Aerospace Research and Development.
13.
EndsleyM. R. (1990b). Predictive utility of an objective measure of situation awareness. In Proceedings of the Human Factors Society 34th Annual Meeting (pp. 41–45). Santa Monica, CA: Human Factors Society.
14.
EndsleyM. R. (1993). A survey of situation awareness requirements in air-to-air combat fighters. International Journal of Aviation Psychology, 3, 157–168.
15.
EndsleyM. R. (1995a). Measurement of situation awareness in dynamic systems. Human Factors, 37, 65–84.
16.
EndsleyM. R. (1995b). Toward a theory of situation awareness in dynamic systems. Human Factors, 37, 32–64.
17.
EndsleyM. R. (2000). Direct measurement of situation awareness: Validity and use of SAGAT. In EndsleyM. R.GarlandD. J. (Eds.), Situation awareness analysis and measurement (pp. 147–174). Mahwah, NJ: Lawrence Erlbaum.
18.
EndsleyM. R. (2018a). Combating information attacks in the age of the internet: New challenges for cognitive engineering. Human Factors, 60, 1081–1094.
19.
EndsleyM. R. (2018b). Expertise and situation awareness. In EricssonK. A.HoffmanR. R.KozbeltA.WilliamsA. M. (Eds.), Cambridge handbook of expertise and expert performance (2nd ed., pp. 714–744). Cambridge, UK: Cambridge University Press.
20.
EndsleyM. R. (in press). A systematic review and meta-analysis of direct, objective measures of SA: A comparison of SAGAT and SPAM. Human Factors.
21.
EndsleyM. R.BolstadC. A. (1994). Individual differences in pilot situation awareness. International Journal of Aviation Psychology, 4, 241–264.
22.
EndsleyM. R.GarlandD. J. (Eds.). (2000). Situation awareness analysis and measurement. Mahwah, NJ: Lawrence Erlbaum.
23.
EndsleyM. R.JonesD. G. (2012). Designing for situation awareness: An approach to human-centered design (2nd ed.). London, England: Taylor & Francis.
24.
EndsleyM. R.JonesW. M. (1997). Situation awareness, information dominance, and information warfare (AL/CF-TR-1997–0156). Wright-Patterson AFB, OH: United States Air Force Armstrong Laboratory.
25.
EndsleyM. R.SelconS. J.HardimanT. D.CroftD. G. (1998). A comparative evaluation of SAGAT and SART for evaluations of situation awareness. In Proceedings of the Human Factors and Ergonomics Society Annual Meeting (pp. 82–86). Santa Monica, CA: Human Factors and Ergonomics Society.
26.
EndsleyM. R.SollenbergerR.SteinE. (2000). Situation awareness: A comparison of measures. In Proceedings of the Human Performance, Situation Awareness and Automation: User-Centered Design for the New Millennium (pp. 15–19). Savannah, GA: SA Technologies.
27.
FlemingS. M.LauH. C. (2014). How to measure metacognition. Frontiers in Human Neuroscience, 8, Article 443.
28.
GatsoulisY.VirkG. S.Dehghani-SanijA. A. (2010). On the measurement of situation awareness for effective human-robot interaction in teleoperated systems. Journal of Cognitive Engineering and Decision Making, 4, 69–98.
29.
GonzalezC.WimisbergJ. (2007). Situation awareness in dynamic decision making: Effects of practice and working memory. Journal of Cognitive Engineering and Decision Making, 1, 56–74.
30.
GrimaldiP.LauH.BassoM. A. (2015). There are things that we know that we know, and there are things that we do not know we do not know: Confidence in decision-making. Neuroscience & Biobehavioral Reviews, 55, 88–97.
31.
GutzwillerR. S.CleggB. A. (2013). The role of working memory in levels of situation awareness. Journal of Cognitive Engineering and Decision Making, 7, 141–154.
32.
HamiltonK.MancusoV.MohammedS.TeslerR.McNeeseM. (2017). Skilled and unaware: The interactive effects of team cognition, team metacognition, and task confidence on team performance. Journal of Cognitive Engineering and Decision Making, 11, 382–395.
33.
HaussY.EyferthK. (2003). Securing future ATM-concepts’ safety by measuring situation awareness in ATC. Aerospace Science and Technology, 7, 417–427.
34.
HoggD. N.Folles⊘K.Strand-VoldenF.TorralbaB. (1995). Development of a situation awareness measure to evaluate advanced alarm systems in nuclear power plant control rooms. Ergonomics, 38, 2394–2413.
35.
JippM.AckermanP. L. (2016). The impact of higher levels of automation on performance and situation awareness: A function of information-processing ability and working-memory capacity. Journal of Cognitive Engineering and Decision Making, 10, 138–166.
36.
JonesD. G.EndsleyM. R. (1996). Sources of situation awareness errors in aviation. Aviation, Space, and Environmental Medicine, 67, 507–512.
37.
JonesD. G.EndsleyM. R. (2000a). Can real-time probes provide a valid measure of situation awareness? In Proceedings of the Human Performance, Situation Awareness and Automation: User-Centered Design for the New Millennium Conference (pp. 245–250). Savannah, GA: SA Technologies.
38.
JonesD. G.EndsleyM. R. (2000b). Overcoming representational errors in complex environments. Human Factors, 42, 367–378.
39.
JonesD. G.EndsleyM. R. (2004). Using real time probes for measuring situation awareness. International Journal of Aviation Psychology, 14, 343–367.
40.
KrugerJ.DunningD. (1999). Unskilled and unaware of it: How difficulties in recognizing one’s own incompetence lead to inflated self-assessments. Journal of Personality and Social Psychology, 77, 1121–1134.
41.
LeeJ. D. (1999). Measuring driver adaptation to in-vehicle information systems: Disassociation of subjective and objective situation awareness measures. In Proceedings of the Human Factors and Ergonomics Society Annual Meeting (pp. 992–996). Los Angeles, CA: SAGE.
42.
LiX.SandersonP.MemisevicR.WongW. (2007). Adapting situational awareness measures for hydropower display evaluations. In Proceedings of the Human Factors and Ergonomics Society Annual Meeting (pp. 210–214). Los Angeles, CA: SAGE.
43.
LichaczF. M. (2008). Augmenting understanding of the relationship between situation awareness and confidence using calibration analysis. Ergonomics, 51, 1489–1502.
44.
LichaczF. M. (2009). Calibrating situation awareness and confidence within a multinational coalition operation. Military Psychology, 21, 412–426.
45.
LichaczF. M.CainB.PatelS. (2003). Calibration of confidence in situation awareness queries. In Proceedings of the Human Factors and Ergonomics Society Annual Meeting (pp. 222–226). Los Angeles, CA: SAGE.
46.
LoJ. C.SehicE.BrookhuisK. A.MeijerS. A. (2016). Explicit or implicit situation awareness? Measuring the situation awareness of train traffic controllers. Transportation Research Part F: Traffic Psychology and Behaviour, 43, 325–338.
47.
LoftS.BowdenV.BraithwaiteJ.MorrellD. B.HufS.DursoF. T. (2015). Situation awareness measures for simulated submarine track management. Human Factors, 57, 298–310.
48.
LoftS.MorrellD. B.HufS. (2013). Using the situation present assessment method to measure situation awareness in simulated submarine track management. International Journal of Human Factors and Ergonomics, 2, 33–48.
49.
LoftS.MorrellD. B.PontonK.BraithwaiteJ.BowdenV.HufS. (2016). The impact of uncertain contact location on situation awareness and performance in simulated submarine track management. Human Factors, 58, 1052–1068.
50.
MatthewsM. D.BealS. A. (2002). Assessing situation awareness in field training exercises. West Point, NY: U.S. Military Academy, Office of Military Psychology and Leadership.
51.
McGuinnessB. (2004). Quantitative analysis of situational awareness (QUASA): Applying signal detection theory to true/false probes and self-ratings. In Proceedings of the Command and Control Research and Technology Symposium (pp. 1–14). Bristol, UK: BAE Systems Advanced Technology Centre.
52.
McGuinnessB.FoyJ. L. (2000). A subjective measure of situation awareness: The crew awareness rating scale. In Proceedings of the First Human Performance, Situation Awareness and Automation Conference (pp. 286–291). Savannah, GA: SA Technologies.
53.
MorganC. A.ChiappeD. L.KrautJ.StrybelT. Z.VuK. P. L. (2012). An investigation into the potential sources of interference in SA probe techniques. In LandryS. (Ed.), Advances in Human Aspects of Aviation (pp. 623–631). Boca Raton, FL: CRC Press.
54.
NaderpourM.LuJ.ZhangG. (2016). A safety-critical decision support system evaluation using situation awareness and workload measures. Reliability Engineering & System Safety, 150, 147–159.
55.
NickersonR. S. (1998). Confirmation bias: A ubiquitous phenomenon in many guises. Review of General Psychology, 2, 75–220.
56.
NisbettR. E.WilsonT. D. (1977). Telling more than we can know: Verbal reports on mental processes. Psychological Review, 84, 231–259.
57.
PierceC. S.JastrowJ. (1885). On small differences in sensation. Memoirs of the National Academy of Sciences, 3, 73–83.
58.
PierceR. S. (2012). The effect of SPAM administration during a dynamic simulation. Human Factors, 54, 838–848.
59.
PierceR. S.StrybelT. Z.VuK.-P. L. (2008). Comparing situation awareness measurement techniques in a low fidelity air traffic control simulation. In Proceedings of the 26th International Congress of the Aeronautical Sciences (ICAS) (pp. 1–8). Anchorage, AS: International Congress of the Aeronautical Sciences.
60.
PriceT.TenanM.HeadJ.MaslinW.LaFiandraM. (2016). Acute stress causes over confidence in situation awareness. In Proceedings of the 2016 IEEE International Multi-Disciplinary Conference on Cognitive Methods in Situation Awareness and Decision Support (CogSIMA) (pp. 1–6). New York, NY: IEEE.
61.
PuuskaS.RummukainenL.TimonenJ.LääperiL.KlemettiM.OksamaL.VankkaJ. (2018). Nationwide critical infrastructure monitoring using a common operating picture framework. International Journal of Critical Infrastructure Protection, 20, 28–47.
62.
RoseJ.BearmanC.DorrianJ. (2018). The Low-Event Task Subjective Situation Awareness (LETSSA) technique: Development and evaluation of a new subjective measure of situation awareness. Applied Ergonomics, 68, 273–282.
63.
RousseauR.TremblayS.BanburyS.BretonR.GuitouniA. (2010). The role of metacognition in the relationship between objective and subjective measures of situation awareness. Theoretical Issues in Ergonomics Science, 11, 119–130.
64.
SætrevikB. (2012). A controlled field study of situation awareness measures and heart rate variability in emergency handling teams. In Proceedings of the Human Factors and Ergonomics Society Annual Meeting (pp. 2006–2010). Los Angeles, CA: SAGE.
65.
SalmonP. M.StantonN. A.WalkerG. H.JenkinsD.LadvaD.RaffertyL.YoungM. (2009). Measuring situation awareness in complex systems: Comparison of measures study. International Journal of Industrial Ergonomics, 39, 490–500.
66.
SalmonP. M.StantonN. A.YoungK. L. (2011). Situation awareness on the road: Review, theoretical and methodological issues and future directions. Theoretical Issues in Ergonomics Science, 13, 472–492.
67.
SarterN. B.WoodsD. D. (1991). Situation awareness: A critical but ill-defined phenomenon. International Journal of Aviation Psychology, 1, 45–57.
68.
SchusterD.KeeblerJ. R.JentschF.ZunigaJ. (2012). Comparison of SA measurement techniques in a human-robot team task. In Proceedings of the Human Factors and Ergonomics Society Annual Meeting (pp. 1713–1717). Los Angeles, CA: SAGE.
69.
SelconS. J.TaylorR. M. (1990). Evaluation of the situational awareness rating technique (SART) as a tool for aircrew systems design. In Situational awareness in aerospace operations (AGARD-CP-478) (pp. 5/1–5/8). Neuilly Sur Seine, France: North Atlantic Treaty Organization-Advisory Group for Aerospace Research and Development.
70.
SelconS. J.TaylorR. M.KoritsasE. (1991). Workload or situational awareness? TLX vs. SART for aerospace systems design evaluation. In Proceedings of the Human Factors Society 35th Annual Meeting (pp. 62–66). Santa Monica, CA: Human Factors Society.
71.
SelkowitzA. R.LakhmaniS. G.ChenJ. Y. C. (2017). Using agent transparency to support situation awareness of the autonomous squad member. Cognitive Systems Research, 46, 13–25.
72.
SethumadhavanA. (2011). Knowing what you know: The role of meta-situation awareness in predicting situation awareness. In Proceedings of the Human Factors and Ergonomics Society Annual Meeting (pp. 360–364). Los Angeles, CA: SAGE.
73.
ShipsteadZ.LindseyD. R.MarshallR. L.EngleR. W. (2014). The mechanisms of working memory capacity: Primary memory, secondary memory, and attention control. Journal of Memory and Language, 72, 116–141.
74.
SorensenL. J.StantonN. A. (2011). Is SA shared or distributed in team work? An exploratory study in an intelligence analysis task. International Journal of Industrial Ergonomics, 41, 677–687.
75.
StraterL. D.EndsleyM. R.PlebanR. J.MatthewsM. D. (2001). Measures of platoon leader situation awareness in virtual decision making exercises (Research Report No. 1770). Alexandria, VA: Army Research Institute.
76.
StrybelT. Z.VuK.-P. L.DwyerJ. P.KraftJ.NgoT. K.ChambersV.GarciaF. P. (2007). Predicting perceived situation awareness of low altitude aircraft in terminal airspace using probe questions. In Proceedings of the International Conference on Human-Computer Interaction (pp. 939–948). New York, NY: Springer.
77.
StrybelT. Z.VuK.-P. L.KraftJ.MinakataK. (2008). Assessing the situation awareness of pilots engaged in self spacing. In Proceedings of the Human Factors and Ergonomics Society Annual Meeting (pp. 11–15). Los Angeles, CA: SAGE.
78.
SulistyawatiK.WickensC. D.ChuiY. P. (2009). Exploring the concept of team situation awareness in a simulated air combat environment. Journal of Cognitive Engineering and Decision Making, 3, 309–330.
79.
SulistyawatiK.WickensC. D.ChuiY. P. (2011). Prediction in situation awareness: Confidence bias and underlying cognitive abilities. International Journal of Aviation Psychology, 21, 153–174.
80.
TaberM. J.McCabeJ.KleinR. M.PelotR. P. (2013). Development and evaluation of an offshore oil and gas Emergency Response Focus Board. International Journal of Industrial Ergonomics, 43, 40–51.
81.
TrapsilawatiF.WickensC. D.QuX.ChenC.-H. (2016). Benefits of imperfect conflict resolution advisory aids for future air traffic control. Human Factors, 58, 1007–1019.
82.
Van den BeukelA. P.van der VoortM. C. (2017). How to assess driver’s interaction with partially automated driving systems—A framework for early concept assessment. Applied Ergonomics, 59, 302–312.
83.
VidulichM. A. (1989). The use of judgment matrices in subjective workload assessment: The subjective workload dominance (SWORD) technique. In Proceedings of the Human Factors Society 33rd Annual Meeting (pp. 1406–1410). Santa Monica, CA: Human Factors Society.
84.
WalkerG. H.StantonN. A.YoungM. S. (2006). The ironies of vehicle feedback in car design. Ergonomics, 49, 161–179.
85.
WalkerG. H.StantonN. A.YoungM. S. (2008). Feedback and driver situation awareness (SA): A comparison of SA measures and contexts. Transportation Research Part F: Traffic Psychology and Behaviour, 11, 282–299.
86.
WickensC. D. (2008). Multiple resources and mental workload. Human Factors, 50, 449–455.
87.
WillemsB. F.HeineyM. (2001). Real-time assessment of situation awareness of air traffic control specialists on operational host computer system and display system replacement hardware. In Proceedings of the USA/Europe Air Traffic Management R&D Seminar (pp. 1–11). Santa Fe, NM: Federal Aviation Administration/Eurocontrol.
88.
YerkesR. M.DodsonJ. D. (1908). The relation of strength of stimulus to rapidity of habit-formation. The Journal of Comparative Neurology, 18, 459–482.
89.
YeungN.SummerfieldC. (2012). Metacognition in human decision-making: Confidence and error monitoring. Philosophical Transactions of the Royal Society B: Biological Sciences, 367, 1310–1321.
