Several methods of measuring the situation awareness (SA) of a human who is teleoperating a robot are compared for the domain of urban search and rescue (USAR), to identify those that have the greatest potential for developing more reliable and accurate ways of measuring SA for effectively operating robot systems. Such comparative studies are essential because SA has been identified as a key human factors issue since post-September 11, 2001, operations for effective task performance in (tele)robotics. This paper discusses the pros and cons of the main aspects that need to be included in developing reliable SA measurement methods from the related domain of air traffic control, in which SA has a long research history. These methods have been adopted and modified accordingly to address needs of teleoperating robot systems and have been tested in a realistic USAR simulated scenario developed especially for performing these assessments. The results were compared against each other as well as with existing measures. A new method of measuring task performance that is more appropriate for USAR is also presented and tested within the comparative studies.
Get full access to this article
View all access options for this article.
References
1.
AdamsM. J.TenneyY. T.PewR. W.(1995).Situation awareness and the cognitive management of complex systemsHuman Factors, 37, 85–104
2.
BellH. H.LyonD. R.(2000).Using observer ratings to assess situation awareness In EndsleyM. R.GarlandD. J. (Eds.), Situation awareness analysis and measurement (pp. 129–146). Mahwah, NJ: Erlbaum
BlackburnM. R.EverettH. R.LairdR. T.(2002).After action report to the Joint Program Office: Center for the Robotic Assisted Search and Rescue (CRASAR) related efforts at the World Trade Center (Tech. Document 3141). San Diego, CASpace and Naval Warfare Systems Center
5.
BrickmanB. J.HettingerL. J.RoeM. M.StautbergD. K.VidulichM. A.HaasM. W.ShawR. L.(1995).An assessment of situation awareness in an air combat simulation: The Global Implicit Measurement Approach In GarlandD. J.EndsleyM. R. (Eds.), Experimental analysis and measurement of situation awareness: Proceedings of an international conference (pp. 339–344). Daytona Beach, FL: Embry-Riddle Aeronautical University Press
6.
7.
BurkeJ. L.MurphyR. R.CoovertM.RiddleD.(2004).Moonlight in Miami: A field study of human-robot interaction in the context of an urban search and rescue disaster response training exerciseHuman-Computer Interaction, 85–116 (Special Issue on Human-Robot Interaction)
8.
BurkeJ. L.MurphyR. R.RogersE.LumelskyV. J.ScholtzJ.(2004).Final report for the DARPA/NSF interdisciplinary study on human-robot interactionIEEE Transactions on Systems, Man, and Cybernetics, 34, 103–112
9.
CasperJ.MurphyR.(2003).Human-robot interactions during the robot-assisted urban search and rescue response at the World Trade CenterIEEE Transactions on Systems, Man, and Cybernetics—Part B: Cybernetics, 33, 367–385
10.
11.
DennehyK.(1997).Cranfield-situation awareness scale: Users manual.Bedfordshire, UK: Cranfield University, College of Aeronautics, Applied Psychology Unit
12.
DursoF. T.HackworthC. A.TruittT. R.CrutchfieldJ.NikolicD.ManningC. A.(1998).Situation awareness as a predictor of performance for en route air traffic controllersAir Traffic Control Quarterly, 6(1), 1–20
13.
EdgarG. K.EdgarH. E.CurryM. B.(2003).Using signal detection theory to measure situation awareness in command and control In Proceedings of the Human Factors and Ergonomics Society 47th Annual Meeting (pp. 2019–2023). Santa Monica, CA: SAGE Publications
14.
EdgarG. K.SmithA. J.StoneH. E.BeethamD. L.PritchardC.(2000, December).QUASA: QUantifying and Analysing Situational Awareness. Paper presented at the IMCD People in Digitized Command and Control Symposium, RMCS Shrivenham, UK
15.
EndsleyM. R.(1988).Situation Awareness Global Assessment Technique (SAGAT) In Proceedings of the IEEE 1988 National Aerospace and Electronics Conference (NAECON, pp. 789–795). Piscataway, NJ: IEEE
16.
EndsleyM. R.(1990a).A methodology for the objective measurement of situation awareness In Situational awareness in aerospace operations (AGARD-CP-478, pp. 1/1–1/9). Neuilly-sur-Seine, France: NATO-AGARD
17.
EndsleyM. R.(1990b).Situation awareness in dynamic human decision making: Theory and measurement. Unpublished doctoral dissertation, University of Southern California, Los Angeles, CA
18.
EndsleyM. R.(1995a).Measurement of situation awareness in dynamic systemsHuman Factors, 37, 65–84
19.
EndsleyM. R.(1995b).Toward a theory for situation awareness in dynamic systemsHuman Factors, 37, 32–64
20.
EndsleyM. R.(2000a).Direct measurement of situation awareness: Validity and use of SAGAT In EndsleyM. R.GarlandD. J. (Eds.), Situation awareness analysis and measurement (pp. 147–173). Mahwah, NJ: Erlbaum
21.
EndsleyM. R.(2000b).Theoretical underpinnings of situation awareness: A critical review In EndsleyM. R.GarlandD. J. (Eds.), Situation awareness analysis and measurement (pp. 3–32). Mahwah, NJ: Erlbaum
22.
EndsleyM. R.GarlandD. J. (Eds.). (2000).Situation awareness analysis and measurement.Mahwah, NJErlbaum
23.
FieldA.(2005).Discovering statistics using SPSS (2nd ed.). LondonSAGE Publications
24.
25.
26.
GatsoulisY.ChochlidakisI.VirkG. S.(2005).Design toolset for realising robotic systems. In ArmadaM.de SantosP. González (Eds.), Climbing and walking robots: Proceedings of the 7th International Conference CLAWAR 2004 (pp. 285–304). New York: Springer
27.
GatsoulisY.ChochlidakisI.VirkG. S.(2004, September).A software framework for the design and support of mass market CLAWAR machines. Paper presented at the IEEE International Conference on Mechatronics and Robotics (MECHROB′04), Aachen, Germany
GerkeyB. P.VaughanR. T.HowardA.(2003).The Player/Stage project: Tools for multi-robot and distributed sensor systems. In Proceedings of the International Conference on Advanced Robotics (ICAR 2003, pp. 317–323). Retrieved December 22, 2009, from http://robotics.usc.edu/∼gerkey/research/final_papers/icar03-player.pdf
30.
HaussY.EyferthK.(2001, December).Evaluation of a multi-sector-planner concept: SALSA—A new approach to measure situation awareness in ATC. Paper presented at the 4th USA/Europe Air Traffic Management RAD Seminar, Santa Fe, NM
31.
HughesS.LewisM.(2004).Robotic camera control for remote exploration In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 511–517). New York: ACM
32.
HughesS.LewisM.(2005).Task-driven camera operations for robotic explorationIEEE Transactions on Systems, Man and Cybernetics—Part A: Systems and Humans, 35, 513–522
33.
JacoffA.WeissB.MessinaE.TadokoroS.NakagawaY.(2003, October).Test arenas and performance metrics for urban search and rescue robots. Paper presented at the IEEE/RSJ International Conference on Intelligent Robots and Systems, Las Vegas, NV
34.
35.
36.
JonesD. G.(2000).Subjective measures of situation awareness In EndsleyM. R.GarlandD. J. (Eds.), Situation awareness analysis and measurement (pp. 113–128). Mahwah, NJ: Erlbaum
37.
JonesD. G.EndsleyM. R.(2004).Use of real-time probes for measuring situation awarenessInternational Journal of Aviation Psychology, 14, 343–367
38.
JonesD. G.KaberD. B.(2005).Situation awareness measurement and the Situation Awareness Global Assessment Technique In StantonN.HedgeA.BrookhuisK.SalasE.HendrickH. W. (Eds.), Handbook of human factors and ergonomics methods (pp. 42/41–42/48). Boca Raton, FL: CRC Press
39.
KaberD. B.OnalE.EndsleyM. R.(1998).Level of automation effects on telerobot performance and human operator situation awareness and subjective workload In ScerboM. W.MoulouaM. (Eds.), Automation technology and human performance: Current research and trends (pp. 165–170). Mahwah, NJ: Erlbaum
40.
KaberD. B.OnalE.EndsleyM. R.(2000).Design of automation for telerobots and the effect on performance, operator situation awareness, and subjective workloadHuman Factors and Ergonomics in Manufacturing, 10, 409–430
41.
KaberD. B.RileyJ. M.Sheik-NainarM. A.HyattJ. R.ReynoldsJ. P.(2006).Assessing infantry soldier situation awareness in virtual environment-based training of urban terrain operations In Proceedings of the IEA 2006–16th World Congress on Ergonomics [CD-ROM]. New York: Elsevier
42.
KaberD. B.RileyJ. M.ZhouR.DraperJ. V.(2000).Effects of visual interface design and control mode and latency on performance, telepresence, and workload in a teleoperation task In Proceedings of the IEA 2000/HFES 2000 Congress (Vol. 1., pp. 503–506). Santa Monica, CA: SAGE Publications
43.
KaberD. B.WrightM. C.(2003).Automation-state changes and sensory cueing in telerobot control In Proceedings of the XVth Triennial Congress of the International Ergonomics Association (CD-ROM, Vol. 1, Computers and Communications). Seoul, Korea: The Ergonomics Society of Korea
44.
KaberD. B.WrightM. C.Sheik-NainarM.(2006).Multimodal interface design for adaptive automation of a human-robot systemInternational Journal of Human-Computer Studies, 64, 527–540
45.
LikertR.(1932).A technique for the measurement of attitudesArchives of Psychology, 22(140), 1–55
46.
47.
MatthewsM. D.PlebanR. J.EndsleyM. R.StraterL. D.(2000).Measures of infantry situation awareness for a virtual MOUT environment. In Proceedings of the Human Performance, Situation Awareness and Automation: User Centered Design for the New Millennium Conference—October 2000. Retrieved December 23, 2009, from http://www.satechnologies.com/Papers/pdf/HPSAA2000-armySA.pdf
48.
McGuinnessB.(1999).Situational awareness and the Crew Awareness Rating Scale (CARS) In Proceedings of the 1999 Avionics Conference (ERA Technology Report 99–0815, paper 4.3). Leatherhead, UK: ERA Technology
49.
McGuinnessB.(2004).Quantitative analysis of situational awareness (QUASA): Applying signal detection theory to true/false probes and self-ratings In Proceedings of the 9th International Command and Control Research and Technology Symposium (pp. 159–178). Washington, DC: U.S. Department of Defence Command and Control Research Program
50.
MunizE.StoutR.BowersC. A.SalasE.(1998).A methodology for measuring team situational awareness: Situational linked indicators adapted to novel tasks (SALIANT) In RTO Human Factors and Medicine Symposium on Collaborative Crew Performance in Complex Operational Systems (Report No. RTO-MP-4; pp. 11/1–11/8). Neuilly-Sur-Seine, France: NATO Research and Technology Organization
51.
52.
NyquistH.(1928).Certain topics in telegraph transmission theoryTransactions of the American Institute of Electrical Engineers, 47, 617–644
53.
PewR. W.(2000).The state of situation awareness measurement: Heading toward the next century In EndsleyM. R.GarlandD. J. (Eds.), Situation awareness analysis and measurement (pp. 33–47). Mahwah, NJ: Erlbaum
54.
PritchettA. R.HansmanR. J.JohnsonE. N.(1996, November).Use of testable responses for performance-based measurement of situation awareness. Paper presented at the International Conference on Experimental Analysis and Measurement of Situation Awareness, Daytona Beach, FL. Retrieved December 29, 2009, from http://web.mit.edu/aeroastro/www/labs/ASL/SA/sa.html
55.
ReichardK. M.(2004).Integrating self-health awareness in autonomous systemsRobotics and Autonomous Systems, 49, 105–112
56.
ReichardK. M.CrowE. C.(2002).Self-awareness, monitoring and diagnosis for autonomous vehicle operations In AUVSI Unmanned Systems Proceedings [CD-ROM]. Arlington, VA: Association for Unmanned Vehicle Systems International
57.
RileyJ. M.(2001).The utility of measures of attention and situation awareness for quantifying telepresence. PhD Thesis, Department of Industrial & Systems Engineering, Mississippi State University, Starkville, MS
58.
RileyJ. M.KaberD. B.Sheik-NainarM. A.EndsleyM. R.(2008).Enhancing situation awareness training in virtual reality through measurement and feedback In SchmorrowD.CohnJ.NicholsonD. (Eds.), The PSI handbook of virtual environments for training and education: Developments for the military and beyond (pp. 326–347). London: Greenwood
59.
RileyJ. M.StraterL. D.(2006).Effects of robot control mode on situation awareness and performance in a navigation task In Proceedings of the Human Factors and Ergonomics Society 50th Annual Meeting (pp. 540–544). Santa Monica, CA: SAGE Publications
60.
RileyJ. M.StraterL. D.SethumadhavanA.DavisF.TharanathanA.KokiniC.(2008).Performance and situation awareness effects in collaborative robot control with automation In Proceedings of the Human Factors and Ergonomics Society 52nd Annual Meeting (pp. 242–246). Santa Monica, CA: SAGE Publications
61.
SarterN. B.WoodsD. D.(1995).How in the world did we ever get into that mode? Mode error and awareness in supervisory controlHuman Factors, 37, 5–19
ScholtzJ.YoungJ.DruryJ. L.YancoH. A.(2004).Evaluation of human-robot interaction awareness in search and rescue In Proceedings of the IEEE International Conference on Robotics and Automation (ICRA 2004, Vol. 1, pp. 2327–2332). Piscataway, NJ: IEEE
64.
ScholtzJ. C.AntonishekB.YoungJ.(2005).Implementation of a situation awareness assessment tool for evaluation of human-robot interfacesIEEE Transactions on Systems, Man and Cybernetics—Part A: Systems and Humans, 35, 450–459
65.
ShannonC. E.(1949).Communication in the presence of noiseProceedings of the Institute of Radio Engineers, 37(1), 10–21
SteinfeldA.FongT.KaberD. B.LewisM.ScholtzJ.SchultzA.GoodrichM.(2006).Common metrics for human-robot interaction In Proceedings of the 1st ACM SIGCHI/SIGART Conference on Human-Robot Interaction (pp. 33–40). New York: ACM
68.
TaylorR. M.(1990).Situational Awareness Rating Technique (SART): The development of a tool for aircrew systems design In Situational awareness in aerospace operations (AGARD-CP-478, pp. 3.1–3.17). Neuilly-sur-Seine, France: NATO Research and Technology Organization
69.
TaylorR. M.ShadrakeR.HaughJ.BuntingA.(1996).Situational awareness, trust and compatibility: Using cognitive mapping techniques to investigate the relationships between important cognitive system variables In Situation awareness: Limitations and enhancement in the aviation environment, Proceedings of AGARD AMP Symposium (AGARD-CP-575, pp. 6–1–6–14). Neuilly-sur-Seine, France: NATO
70.
71.
VidulichM. A.(1992).Measuring situation awareness In Proceedings of the Human Factors and Ergonomics Society 36th Annual Meeting (pp. 40–41). Santa Monica, CA: SAGE Publications
72.
VidulichM. A.HughesE. R.(1991).Testing a subjective metric of situation awareness In Proceedings of the Human Factors and Ergonomics Society 35th Annual Meeting (pp. 1307–1311). Santa Monica, CA: SAGE Publications
YancoH. A.DruryJ.(2004).“Where am I?” Acquiring situation awareness using a remote robot platform In 2004 IEEE International Conference on Systems, Man and Cybernetics (Vol. 3, pp. 2835–2840). Piscataway, NJ: IEEE
75.
YancoH. A.DruryJ. L.ScholtzJ.(2004).Beyond usability evaluation: Analysis of human-robot interaction at a major robotics competitionHuman-Computer Interaction, 19, 117–149
