The study applied the Cochran-Weiss-Shanteau (CWS) index, a behavioral-based measure of expertise that integrates discrimination and consistency. Larger CWS scores are indicative of better evaluation, i.e., greater discrimination and consistency. CWS was used to assess the performance of controllers operating in high-fidelity simulations of air traffic control (ATC). Large CWS scores were associated with superior performance, e.g., fewer separation errors. The CWS indices were also sensitive to changes in task complexity and controller efficiency, further validating the index. This research extended CWS to real controllers operating in high-fidelity simulations of ATC.
Get full access to this article
View all access options for this article.
References
1.
CochranW. G. (1943). The comparison of different scales of measurement for experimental results. Annals of Mathematical Statistics, 14, 205–216.
2.
GilbertT. F. (1978). Human competence. Engineering worthy performance. New York: McGraw-Hill.
3.
EttensonR. (1984). A schematic approach to the examination of the search for and use of information in expert decision making. Unpublished doctoral dissertation, Kansas State University.
4.
FrielB. M.ThomasR. P.ShanteauJ.RaackeJ., (in preparation). CWS applied to an air traffic control simulation task (CTEAM).
5.
HammondK. R. (1996). Human judgment and social policy. New York: Oxford University Press.
6.
HollnagelE.CacciabueP. C.HocJ. (1995). Work with technology: Some fundamental issues. In HocJ.CacciabueP. C.HollnagelE., Expertise and technology (pp. 1–15). Hillsdale, N J: Lawrence Erlbaum.
7.
ManningC. A.MillsS.MogilkaH.HedgeJ.BruskiewiczPfleidererE., (in preparation) Prediciton of subjective ratings of air traffic controller performance by computer-derived measures and behavioral observations.
8.
NagyR. H. (1977). How are personnel selection decisions made? An analysis of decision strategies in a simulated personnel selection. Unpublished doctoral dissertation, Kansas State University.
9.
PhelpsR. H.ShanteauJ. (1978). Livestock judges: How much information can an expert use? Organizational Behavior and Human Performance, 21, 209–219.
10.
ShanteauJ. (in press). What does it mean when experts disagree? In Klein, et al., To appear in: Naturalistic decision making.
11.
ShanteauJ.WeissD. J.ThomasR. P.PoundsJ. (in press). Performance-based Assessment of Expertise: How can you tell if someone is an expert? European Journal of Operations Research.
12.
SollenbergerR. L.SteinE. S.GromelskiS. (1997). The development and evaluation of a behaviorally based rating form for assessing air traffic controller performance. (FAA Technical Note). Atlantic City: William J. Hughes Technical Center. DOT/FAA/CT-TN96/16.
13.
ThomasR. P.PoundsJ.ShanteauJ. (in press). Evaluation of a performance-based measure of expertise in a dynamic complex environment. Federal Aviation Administration.
