It is a current trend in aviation to use categories of technical (e.g., knowledge) and nontechnical skills (e.g., situation awareness) to assess airline pilots’ performance. Several studies have revealed large disagreement between assessors when airline professionals use these categories to assess the performance of their peers. The aim of the present study is to investigate whether the categories themselves are at the source of disagreement. We explore the reasoning of flight examiners who assess an engine fire scenario in pairs. The results provide insight into the overlap of topics that constitute certain categories. Implications are drawn in regards to the use of assessment categories and their influence on pilot performance assessment.
CharmazK. (2008). Grounded theory. In SmithJ. A. (Ed.), Qualitative psychology: A practical guide to research methods (pp. 81-110). London: SAGE.
2.
ChauvinC.ClostermannJ. P.HocJ.- M. (2008). Situation awareness and the decision-making process in a dynamic situation: Avoiding collisions at sea. Journal of Cognitive Engineering and Decision Making, 2(1), 1-23.
3.
CorbinJ.StraussA. L. (2008). Basics of qualitative research: Techniques and procedures for developing grounded theory (3rd ed.). Los Angeles: SAGE.
4.
DekkerS. W. A. (2006). The field guide to understanding human error. Aldershot, UK: Ashgate.
5.
DekkerS. W. A.HollnagelE. (2004). Human factors and folk models. Cognition, Technology, and Work, 6, 79-86.
6.
DekkerS. W. A.HummerdalD. H.SmithK. (2010). Situation awareness: Some remaining questions. Theoretical Issues in Ergonomics Science, 11(1-2), 131-135.
7.
DekkerS. W. A.NyceJ. N.van WinsenR.HenriqsonE. (2010). Epistemological self-confidence in human factors research. Journal of Cognitive Engineering and Decision Making, 4(1), 27-38.
8.
EndsleyM. R. (1995). Toward a theory of situation awareness in dynamic systems. Human Factors, 37(1), 32-64.
9.
FlinR.MartinL.GoetersK. M.HörmannH. J.AmalbertiR.ValotC.NijhuisH. (2003). Development of the NOTECHS (non-technical skills) system for assessing pilots’ CRM skills. Human Factors and Aerospace Safety, 3(2), 95-117.
10.
FlinR.O’ConnorP.CrichtonM. (2008). Safety at the sharp end: A guide to non-technical skills. Aldershot, UK: Ashgate Publishing Ltd.
11.
GlaserB. G.StraussA. L. (1965). Awareness of dying. Chicago: Aldine.
12.
GlaserB. G.StraussA. L. (1967). The discovery of grounded theory: Strategies for qualitative research. Chicago: Aldine.
13.
GonzalezC.WimisbergJ. (2007). Situation awareness in dynamic decision making: Effects of practice and working memory. Journal of Cognitive Engineering and Decision Making, 1(1), 56-74.
14.
GugertyL. J. (1997). Situation awareness during driving: Explicit and implicit knowledge in dynamic spatial memory. Journal of Experimental Psychology: Applied, 3(1), 42-66.
15.
HelmreichR. L.FousheeH. C. (1993). Why crew resource management? Empirical and theoretical bases of human factors training in aviation. In WienerE.KankiB.HelmreichR. (Eds.), Cockpit resource management (pp. 3-45). San Diego, CA: Academic Press.
16.
HollnagelE.WoodsD. D. (2005). Joint cognitive systems: Foundations of cognitive systems engineering. Boca Raton, FL: Taylor & Francis.
17.
JARTEL. (2002). Final Report, JAR TEL, Consolidation of Results, WP7 Draft Report (JAR TEL/WP7/D9_07).
18.
MavinT. J.Dall’AlbaG. (2010, April). A model for integrating technical skills and NTS in assessing pilots’ performance. Paper presented at the 9th International Symposium of the Australian Aviation Psychology Association, Sydney, Australia.
19.
MavinT. J.RothW.- M.DekkerS. W. A. (2013). Understanding variance in pilot performance ratings: Two studies of flight examiners, captains and first officers assessing the performance of peers. Aviation Psychology and Applied Human Factors, 3(2), 53-62.
20.
O’ConnorP.HörmannH. J.FlinR.LodgeM.GoetersK. M., & JARTEL Group. (2002). Developing a method for evaluating crew resource management skills: A European perspective. International Journal of Aviation Psychology, 12(3), 263-285.
21.
OrladyH. W.OrladyL. M. (1999). Human factors in multi-crew flight operations. Burlington, VT: Ashgate.
22.
RothW.- M.MavinT. J. (2013). Assessment of non-technical skills: From measurement to categorization modeled by fuzzy logic. Aviation Psychology and Applied Human Factors, 3(2), 73-82.
23.
SalmonP. M.StantonN. A. (2013). Situation awareness and safety: Contribution or confusion? Situation awareness and safety editorial. Safety Science, 56, 1-5.
24.
SalmonP. M.StantonN. A.WalkerG. H.BaberC.JenkinsD. P.McMasterR.YoungM. S. (2008). What really is going on? Review of situation awareness models for individuals and teams. Theoretical Issues in Ergonomics Science, 9(4), 297-323.
25.
SalmonP.StantonN.WalkerG.GreenD. (2006). Situation awareness measurement: A review of applicability for C4i environments. Applied Ergonomics, 37, 225-238.
26.
WeberD. E.RothW.- M.MavinT. J.DekkerS. W. A. (2013). Should we pursue inter-rater reliability or diversity? An empirical study of pilot performance assessment. Aviation in Focus–Journal of Aeronautical Sciences, 4(2), 34-58.
27.
WhitehurstG. (2013). Dwindling resources: The use of single-case research designs as an efficient alternative for applied aviation research. Aviation Psychology and Applied Human Factors, 3(2), 63-72.
