Integrating cognitive engineering into the systems engineering process requires measurement methodologies that capitalize on variability in cognition and behavior distributed across people and their environment. It is important for cognitive system integration that measures be not only reliable and valid but also unobtrusive and capable of providing predictive and diagnostic information in real time. To achieve these objectives we have developed measures of systems (in our case, small teams of humans) that are based heavily on the automated sequential analysis of communication data. These measures have been mapped to system performance, system change, system process, coordination, and situation awareness and thus are of potential value for cognitive system integration activities.
Get full access to this article
View all access options for this article.
References
1.
BernsteinN. A. (1967). The coordination and regulation of movements. London: Pergamon Press.
2.
BerryM. W. (1992). Large scale singular value computations. International Journal of Supercomputer Applications, 6, 13–49.
3.
BØdkerS. (1996). Applying activity theory to video analysis: How to make sense of video data in HCI. In NardiB. (Ed.), Context and consciousness: Activity theory and human-computer interaction (pp. 147–174). Cambridge, MA: MIT Press.
4.
BolstadC. A.EndsleyM. R. (2000). The effect of task load and shared displays on team situation awareness. In Proceedings of the XIVth Triennial Congress of the International Ergonomics Association and 44th Annual Meeting of the Human Factors and Ergonomics Society (pp. 1.189–1.192). Santa Monica, CA: SAGE Publications.
5.
BolstadC. A.EndsleyM. R. (2003). Measuring shared and team situational awareness in the army's future objective force. In Proceedings of the Human Factors and Ergonomics Society 47th Annual Meeting (pp. 369–373). Santa Monica, CA: SAGE Publications.
6.
BowersC. A.JentschF.SalasE.BraunC. C. (1998). Analyzing communication sequences for team training needs assessment. Human Factors, 40, 672–679.
7.
CarreiraR.CratoJ. M.GoncalvesD.JorgeJ. A. (2004). Evaluating adaptive user profiles for news classification. In Proceedings of the 9th International Conference on Intelligent User Interfaces (pp. 206–212). New York: Association for Computing Machinery Press.
8.
CookeN. J.GormanJ. C.DuranJ. L.TaylorA. R. (2007). Team cognition in experienced command-and-control teams. Journal of Experimental Psychology: Applied, 13, 146–157.
9.
CookeN. J.GormanJ. C.KiekelP. A. (2008). Communication as team-level cognitive processing. In LetskyM.WarnerN.FioreS.SmithC. A. P., Macrocognition in teams (pp. 51–64). Hants., UK: Ashgate.
10.
CookeN. J.GormanJ. C.KiekelP. A.FoltzP.MartinM. (2005). Using team communication to understand team cognition in distributed vs. co-located mission environments (Tech. Rep. for ONR Grant N00014–03–1–0580). Las Cruces, NM: New Mexico State University.
11.
CookeN. J.GormanJ. C.PedersenH. K.WinnerJ. L.DuranJ.TaylorA., et al. (2007). Acquisition and retention of team coordination in command-and-control (Tech. Rep. for AFOSR Grant FA9550–04–1–0234 and AFRL Award FA8650–04–6442). Mesa, AZ: Cognitive Engineering Research Institute.
12.
CookeN. J.GormanJ. C.RoweL. J. (2009). An ecological perspective on team cognition, Team effectiveness in complex organizations: Cross-disciplinary perspectives and approaches (pp. 157–182). London: Taylor & Francis.
13.
CookeN. J.GormanJ. C.WinnerJ. L. (2007). Team cognition. In DursoF.NickersonR.DumaisS.LewandowskyS.PerfectT. (Eds.), Handbook of applied cognition (2nd ed., pp. 239–268). New York: Wiley.
14.
CookeN. J.KiekelP. A.HelmE. (2001). Measuring team knowledge during skill acquisition of a complex task. International Journal of Cognitive Ergonomics, 5, 297–315.
15.
CookeN. J.KiekelP. A.SalasE.StoutR. J.BowersC.Cannon-BowersJ. (2003). Measuring team knowledge: A window to the cognitive underpinnings of team performance. Group Dynamics: Theory Research and Practice, 7, 179–199.
16.
CookeN. J.NevilleK. J.RoweA. L. (1996). Procedural network representations of sequential data. Human-Computer Interaction, 11, 29–68.
17.
CookeN. J.SalasE.Cannon-BowersJ. A.StoutR. (2000). Measuring team knowledge. Human Factors, 42, 151–173.
18.
CookeN. J.ShopeS. M. (2005). Synthetic task environments for teams: CERTT's UAV-STE. In StantonN.HedgeA.BrookhuisK.SalasE.HendrickH. (Eds.), Handbook of human factors and ergonomics methods (pp. 46.41–46.46). Boca Raton, FL: CRC Press.
19.
DeerwesterS.DumaisS. T.FurnasG. W.LandauerT. K.HarshmanR. (1990). Indexing by latent semantic analysis. Journal of the American Society for Information Science, 41, 391–407.
20.
DursoF. D.GronlundS. D. (1999). Situation awareness. In DursoF. T.NickersonR.SchvaneveldtR.DumaisS.ChiM.LindsayS. (Eds.), The handbook of applied cognition (pp. 283–314). New York: Wiley.
21.
EndsleyM. R. (1995a). Measurement of situation awareness in dynamic systems. Human Factors, 37, 65–84.
22.
EndsleyM. R. (1995b). Toward a theory of situation awareness in dynamic systems. Human Factors, 37, 32–64.
23.
EricssonK. A.SimonH. A. (1984). Protocol analysis: Verbal reports as data (Rev. ed.). Cambridge, MA: MIT Press.
24.
GluckK. A.BallJ. T.GunzelmannG.KrusmarkM. A.LyonD. R.CookeN. J. (2005, September). A prospective look at Synthetic Teammate for UAV applications. Invited talk for AIAA Infotech@Aerospace Conference on Cognitive Modeling, Arlington, VA.
25.
GormanJ. C.CookeN. J.AmazeenP. G.WinnerJ. L.DuranJ. L.PedersenH. K., et al. (2007). Knowledge training versus process training: The effects of training protocol on team coordination and performance. In Proceedings of the Human Factors and Ergonomics Society 51st Annual Meeting (pp. 382–386). Santa Monica, CA: SAGE Publications.
26.
GormanJ. C.CookeN. J.PedersenH. P.ConnorO. O.DeJoodeJ. A. (2005). Coordinated awareness of situation by teams (CAST): Measuring team situation awareness of a communication glitch. In Proceedings of the Human Factors and Ergonomics Society 49th Annual Meeting (pp. 274–277). Santa Monica, CA: SAGE Publications.
27.
GormanJ. C.CookeN. J.PedersenH. K.WinnerJ. L.AndrewsD.AmazeenP. G. (2006). Changes in team composition after a break: Building adaptive command-and-control teams. In Proceedings of the Human Factors and Ergonomics Society 49th Annual Meeting (pp. 487–491). Santa Monica, CA: SAGE Publications.
28.
GormanJ. C.CookeN. J.WarnerN. W.WroblewskiE. M. (2007). Tagging macrocognitive processes using communication flow patterns. In Proceedings of the Human Factors and Ergonomics Society 51st Annual Meeting (pp. 410–415). Santa Monica, CA: SAGE Publications.
29.
GormanJ. C.CookeN. J.WinnerJ. L. (2006). Measuring team situation awareness in decentralized command and control systems. Ergonomics, 49, 1312–1325.
30.
GormanJ. C.CookeN. J.WinnerJ. L.DuranJ. L.PedersenH. K.TaylorA. R. (2007). Knowledge training versus process training: The effects of training protocol on team coordination and performance. In Proceedings of the Human Factors and Ergonomics Society 51st Annual Meeting (pp. 382–386). Santa Monica, CA: SAGE Publications.
31.
GormanJ. C.FoltzP. W.KiekelP. A.MartinM. J.CookeN. J. (2003). Evaluation of latent-semantic analysis-based measures of team communications. In Proceedings of the Human Factors and Ergonomics Society 47th Annual Meeting (pp. 424–428). Santa Monica, CA: SAGE Publications.
32.
HakenH. (2004). Synergetics: Introduction and advanced topics. Berlin: Springer.
33.
HakenH.KelsoJ. A. S.BunzH. (1985). A theoretical model of phase transitions in human hand movements. Biological Cybernetics, 51, 347–356.
34.
KiekelP. A. (2004). Developing automatic measures of team cognition using communication data. Unpublished Ph.D. thesis, New Mexico State University, Las Cruces, NM.
35.
KiekelP. A.CookeN. J.FoltzP. W.GormanJ. C.MartinM. J. (2002). Some promising results of communication-based automatic measures of team cognition. In Proceedings of the Human Factors and Ergonomics Society 46th Annual Meeting (pp. 298–302). Santa Monica, CA: SAGE Publications.
36.
LandauerT. K.DumaisS. T. (1997). A solution to Plato's problem: The latent semantic analysis theory of the acquisition, induction, and representation of knowledge. Psychological Review, 104, 211–240.
37.
LandauerT. K.FoltzP. W.LahamD. (1998). An introduction to latent semantic analysis. Discourse Processes, 25, 259–284.
38.
Langan-FoxJ.CodeS.Langfield-SmithK. (2000). Team mental models: Techniques, methods, and analytic approaches. Human Factors, 42, 242–271.
39.
LetskyM.WarnerN.FioreS. M.RosenN.SalasE. (2007, June). Macrocognition in complex team problem solving. Paper presented at 12TH ICCRTS Adapting C2 to the 21st Century, Newport, RI.
40.
LevesqueL. L.WilsonJ. M.WholeyD. R. (2001). Cognitive divergence and shared mental models in software development project teams. Journal of Organizational Behavior, 22, 135–144.
41.
MathieuJ. E.GoodwinG. F.HeffnerT. S.SalasE.Cannon-BowersJ. A. (2000). The influence of shared mental models on team process and performance. Journal of Applied Psychology, 85, 273–283.
42.
NewellA. (1990). Unified theories of cognition. Cambridge, MA: Harvard University Press.
43.
OlsonG. M.HerbslebJ.ReuterH. (1994). Characterizing the sequential structure of interactive behaviors through statistical and grammatical techniques. Human-Computer Interaction, 9, 427–472.
44.
PaleyM. J.LinegangM. P.MorleyR. M. (2002). Using communication data to assess organizational and system effectiveness in future combat systems. In Proceedings of the Human Factors and Ergonomics Society 46th Annual Meeting (pp. 290–294). Santa Monica, CA: SAGE Publications.
45.
PewR. W.MavorA. S. (Eds.) (2007). Human-system integration in the system development process: A new look. Washington, DC: National Academies Press.
46.
SalasE.FioreS. M. (Eds.) (2004). Team cognition: Understanding the factors that drive process and performance. Washington, DC: American Psychological Association.
SarterN.SarterM. (2003). Neuroergonomics: Opportunities and challenges of merging cognitive neuroscience with cognitive ergonomics. Theoretical Issues in Ergonomics Science, 4, 142–150.
50.
SchmidtR. C.BienvenuM.FitzpatrickP. A.AmazeenP. G. (1998). A comparison of intra- and interpersonal interlimb coordination: Coordination breakdowns and coupling strength. Journal of Experimental Psychology: Human Perception and Performance, 24, 884–900.
SchvaneveldtR. W.DursoF. T.DearholtD. W. (1989). Network structures in proximity data. In BowerG. H. (Ed.), The psychology of learning and motivation: Advances in research and theory (Vol. 24, pp. 249–284). New York: Academic Press.
53.
SteinfeldA.FongT.KaberD.LewisM.ScholtzJ.SchultzA., et al. (2006, March). Common metrics for human-robot interaction. Paper presented at the First Annual Human-Robot Interaction Conference, Salt Lake City, UT.
54.
StoutR. J.Cannon-BowersJ. A.SalasE.MilanovichD. M. (1999). Planning, shared mental models, and coordinated performance: An empirical link is established. Human Factors, 41, 61–71.
55.
VicenteK. J. (1999). Cognitive work analysis. Mahwah, NJ: Erlbaum.
56.
Von HolstE. (1973). The behavioral physiology of man and animals. Coral Gables, FL: University of Miami Press. (Original work published 1939).
57.
WeilS. A.FosterP.FreemanJ.CarleyK.DiesnerJ.FranzT.CookeN.ShopeS.GormanJ. (2008). IMAGES: Converging approaches to automated communications-based assessment of team situational awareness. In LetskyM.WarnerN.FioreS.SmithC. A. P. (Eds.), Macrocognition in teams: Theories and methodologies (pp. 277–303). Hants., UK: Ashgate.
