In this paper, we describe an integrative approach to understanding flight crew activity. Our approach combines contemporary innovations in cognitive science theory with a new suite of methods for measuring, analyzing, and visualizing the activities of commercial airline flight crews in interaction with the complex automated systems found on the modern flight deck. Our unit of analysis is the multiparty, multimodal activity system. We installed a variety of recording devices in high-fidelity flight simulators to produce rich, multistream time-series data sets. The complexity of such data sets and the need for manual coding of high-level events make large-scale analysis prohibitively expensive. We break through this analysis bottleneck by using our newly developed integrated software system called ChronoViz, which supports visualization and analysis of multiple sources of time-coded data, including multiple sources of high-definition video, simulation data, transcript data, paper notes, and eye gaze data. Four examples of flight crew activity serve to illustrate the methods, the theory, and the kinds of findings that are now possible in the study of flight crew interaction with flight deck automation.
ClarkA. (2003). Natural-born cyborgs: Minds, technologies and the future of human intelligence. Oxford, UK; Oxford University Press.
3.
ClarkA. (2008). Supersizing the mind: Embodiment, action, and cognitive extension. Oxford, UK: Oxford University Press.
4.
ColeM. (1996). Cultural psychology: A once and future discipline. Cambridge, MA: Belknap Press.
5.
FouseA.WeibelN.HutchinsE.HollanJ. D. (2011). ChronoViz: A System for Supporting Navigation of Time-coded Data. In Extended abstracts of CHI 2011, ACM Conference on Human Factors in Computing Systems (pp. 299–304). New York, NY: ACM Press.
6.
GibbsR. (2006). Embodiment and cognitive science. Cambridge, UK: Cambridge University Press.
7.
GoodwinC. (1994). Professional vision. American Anthropologist, 96, 606–633.
8.
HagedornJ.HailpernJ.KarahaliosK. G. (2008). VCode and VData: Illustrating a new framework for supporting the video annotation workflow. In Proceedings of AVI 2008 (pp. 317–321). New York, NY: ACM Press.
9.
HolderB.HutchinsE. (2001). What pilots learn about autoflight while flying on the line. In Proceedings of the 11th International Symposium on Aviation Psychology (pp. 526–531). Dayton, OH: Wright State University.
10.
HollanJ.HutchinsE.KirshD. (2000). Distributed cognition: toward a new foundation for human-computer interaction research. ACM Transactions on Computer–Human Interaction, 7, 174–196.
11.
HutchinsE. (1995a). Cognition in the wild. Cambridge, MA: MIT Press.
12.
HutchinsE. (1995b). How a cockpit remembers its speeds. Cognitive Science, 19, 265–288.
13.
HutchinsE. (2000). The cognitive consequences of patterns of information flow. Intellectica, 1(30), 53–74.
14.
HutchinsE. (2007). Measuring change in pilots’ conceptual understanding of autoflight. In Proceedings of the 14th International Symposium on Aviation Psychology (pp. 281–286). Dayton, OH: Wright State University.
15.
HutchinsE.KlausenT. (1996). Distributed cognition in an airline cockpit. In EngeströmY.MiddletonD. (Eds.), Cognition and communication at work (pp. 15–34). New York, NY: Cambridge University Press.
16.
HutchinsE.MiddletonC.NewsomeW. (2009). Conceptualizing spatial relations in flight training. In Proceedings of the 15th International Symposium on Aviation Psychology (pp. 384–389). Dayton, OH: Wright State University.
17.
HutchinsE.NomuraS. (2011). Collaborative construction of multimodal utterances. Embodied interaction: Language and body in the material world (pp. 29–43). New York, NY: Cambridge University Press.
18.
HutchinsE.NomuraS.HolderB. (2006). The ecology of language practices in worldwide airline flight deck operations. In Proceedings of the 28th Annual Conference of the Cognitive Science Society (CogSci 2006) (pp. 363–368). New York, NY: Psychology Press.
19.
HutchinsE.PalenL. (1997). Constructing meaning from space. In ResnickL. B.SaljoR.PontecorvoC.BurgeB. (Eds.), Discourse, tools, and reasoning: Essays on situated cognition (pp. 23–40). Heidelberg, Germany: Springer-Verlag.
20.
McNeillD. (2005). Gesture and thought. Chicago, IL: University of Chicago Press.
21.
NardiB. (1996). Context and consciousness: Activity theory and human–computer interaction. Cambridge, MA: MIT Press.
22.
NomuraS.HutchinsE. (2007). the multimodal production of common ground understandings in intercultural flight training. In Proceedings of the 14th International Symposium on Aviation Psychology (ISAP 2007) (pp. 475–480). Dayton, OH: Wright State University.
23.
NomuraS.HutchinsE.HolderB. E. (2006). The uses of paper in commercial airline flight operations. In Proceedings of CSCW ’06, International Conference on Computer Supported Cooperative Work (pp. 249–258). New York, NY: ACM Press.
24.
NormanD. A. (1993). Things that make us smart. Boston, MA: Addison-Wesley.
25.
NüssliM. A. (2011). Dual eye-tracking methods for the study of remote collaborative problem solving. Lausanne, Switzlerland: École Polytechnique Fédérale de Lausanne.
26.
PeaR. D. (1993). Practices of distributed intelligence and designs for education. In SalomonG. (Ed.), Distributed cognitions (pp. 47–87). Cambridge, UK: Cambridge University Press.
27.
PeaR.MillsM.RosenJ.DauberK.EffeslbergW.HoffertE. (2004). The DIVER Project: Interactive digital video repurposing. IEEE Multimedia, 11, 54–61.
28.
StewartJ.GappeneO.Di PaoloE. A. (2010). Enaction: Towards a new paradigm in cognitive science. Cambridge, MA: MIT Press.
29.
SuchmanL. (1987). Plans and situated actions: The problem of human–machine communication. Cambridge, UK: Cambridge University Press.
30.
WeibelN.FouseA.EmmeneggerC.FriedmanW.HutchinsE.HollanJ. D. (2012). Digital pen and paper practices in observational research. In Proceedings of CHI 2012, ACM Conference on Human Factors in Computing Systems (pp. 1331–1340). New York, NY: ACM Press.
31.
WeibelN.FouseA.HutchinsE.HollanJ. D. (2011, February). Supporting an integrated paper-digital workflow for observational research. Paper presented at IUI 2011, International Conference on Intelligent User Interfaces, Palo Alto, CA.
32.
WittenburgP.BrugmanH.RusselA.KlassmannA.SloetjesH. (2006). ELAN: A professional framework for multimodality research. In Proceedings of LREC 2006 [CD-ROM].
33.
ZimmermanP. H.BolhuisJ. E.WillemsenA.MeyerE. S.NoldusL. P. J. J. (2009). The Observer XT: A tool for the integration of and synchronization of multimodal signals. Behavioral Research Methods, 41, 731–735.
