Providing decision support to operators in command and control contexts requires careful assessment of its impacts on task performance. Here we describe a human-in-the-loop experiment using a naval air defense testbed to compare three conditions: 1) a baseline interface (DSSBASE; 2) one that displays the temporal proximity of radar aircraft (DSSTEMP); and 3) one which adds a change history panel (DSSCHEX). Threatevaluation accuracy and cognitive models of participants’ judgments did not significantly differ across conditions. Eye-tracking data showed that DSSTEMP lead to a reduced verification of track attributes. Confusion matrices also differed across conditions: DSSTEMP lead to more errors that are “two-categoriesaway” when erroneously classifying hostile aircraft as non-hostile or uncertain. We conclude that in addition to looking at standard metrics of task accuracy and response times, critical insights can be obtained by assessing how different designs for decision support may alter strategies and cognitive processes.
Get full access to this article
View all access options for this article.
References
1.
ArielyD.ZakayD. (2001). A timely account of the role of duration in decision making. Acta psychologica, 108(2), 187-207.
2.
BacklundL. G.BringJ.SkånérY.StrenderL. E.MontgomeryH. (2009). Improving fast and frugal modeling in relation to regression analysis: Test of 3 models for medical decision making. Medical Decision Making, 29, 140–148.
3.
BrackenB.K.PalmonN.RomeroV.PfautzJ.CookeN.J.2014. A Prototype Toolkit for Sensing and Modeling Individual and Team State. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 58, 949–953.
4.
BrehmerA.BrehmerB. (1988). What have we learned about human judgment from thirty years of policy capturing? In BrehmerB.JoyceC. R. B. (Eds), Human Judgment: The SJT View (pp. 75–114). Amsterdam: North-Holland.
5.
Cannon-BowersJ. A.SalasE. (1998). Making decisions under stress: Implications for individual and team training. Washington, DC: American Psychological Association.
6.
ChalmersB. A. (2003, June). Supporting threat response management in a tactical naval environment. 8th International Command and Control Research and Technology Symposium, Washington, DC.
7.
CookG. J.SwainM. R. (1993). A computerized approach to decision process tracing for decision support system design. Decision Sciences, 24, 931-952.
8.
CookeN. J.DursoF. (2007). Stories of modern technology failures and cognitive engineering successes. CRC Press.
9.
CookseyR. W. (1996). Judgment analysis: Theory, methods, and applications. San Diego, CA: Academic Press.
10.
DhamiM. K.HarriesC. (2001). Fast and frugal versus regression models of human judgement. Thinking and Reasoning, 7, 5–27.
11.
DiricanA.C.GöktürkM.2011. Psychophysiological measures of human cognitive states applied in human computer interaction. Procedia Computer Science, World Conference on Information Technology3, 1361–1367.
12.
EricssonK. A.SimonH. A. (1984). Protocol analysis: Verbal reports as data. Cambridge, MA: MIT Press.
13.
FaircloughS. H. (2009). Fundamentals of physiological computing. Interacting with computers, 21(1), 133-145.
14.
FordJ. K.SchmittN.SchechtmanS. L.HultsB. M.DohertyM. L. (1989). Process tracing methods: Contributions, problems, and neglected research questions. Organizational Behavior and Human Decision Processes, 43, 75-117.
15.
GidlöfK.WallinA.DewhurstR.HolmqvistK. (2013). Using eye tracking to trace a cognitive process: Gaze behaviour during decision making in a natural environment. Journal of Eye Movement Research, 6(1), 1-14.
16.
GonzalezC. (2005). Decision support for real-time, dynamic decision-making tasks. Organizational Behavior and Human Decision Processes, 96, 142–154.
17.
HoffmanR. R.KleinG. L. (2017). Challenges and Prospects for the Paradigm of Naturalistic Decision Making. Journal of Cognitive Engineering and Decision Making, 11(1), 97-104.
18.
LafondD.Roberge-VallièresB.VachonF.TremblayS. (2016). Judgment Analysis in a Dynamic Multitask Environment Capturing Nonlinear Policies Using Decision Trees. Journal of Cognitive Engineering and Decision Making, doi: 10.1177/1555343416661889
19.
LevinD.T.BakerL.J. (2015). Change blindness and inattentional blindness. In FawcettJ.RiskoE.F.KingstoneA. (Eds.), The Handbook of Attention, (pp 199-232), Cambridge, MA: MIT Press.
20.
MatthewsG.Reinerman-JonesL.E.BarberD.J.AbichJ.2015. The Psychometrics of Mental Workload Multiple Measures Are Sensitive but Divergent. Human Factors: The Journal of the Human Factors and Ergonomics Society57, 125–143.
21.
MorrisonJ. G.MarshallS. P.KellyR. T.MooreR. A. (1997). Eye tracking in tactical decision making environments: Implications for decision support evaluation. International Command and Control Research and Technology Symposium, Washington, DC, June 17-20.
22.
McKendrickR.ParasuramanR.AyazH. (2015). Wearable functional near infrared spectroscopy (fNIRS) and transcranial direct current stimulation (tDCS): expanding vistas for neurocognitive augmentation. Frontiers in systems neuroscience, 9, 27.
23.
ParadisS.BenaskeurA.OxenhamM.CutlerP. (2005). Threat evaluation and weapons allocation in network-centric warfare. 7th International Conference on Information Fusion (Vol. 2, pp. 1078–1085).
24.
ParasuramanR.WilsonG. F. (2008). Putting the brain to work: Neuroergonomics past, present, and future. Human Factors: The Journal of the Human Factors and Ergonomics Society, 50(3), 468-474.
25.
QuinlanJ. R. (1993). C4.5: Programs for machine learning. San Francisco, CA: Morgan Kaufmann.
26.
RothrockL.KirlikA. (2003). Inferring rule-based strategies in dynamic judgment tasks: Toward a noncompensatory formulation of the lens model. IEEE Transactions on Systems, Man, and Cybernetics—Part A: Systems and Humans, 33, 58–72.
27.
RothrockL.YinJ. (2008). Integrating compensatory and noncompensatory decision-making strategies in dynamic task environments. In KuglerT.SmithJ. C.ConnollyT.SonY.-J. (Eds.), Decision modeling and behavior in complex and uncertain environments, Part II (pp. 125–141). New York, NY:Springer.
28.
RousseauR.TremblayS.LafondD.VachonF.BretonR. (2007). Assessing temporal support for dynamic decision making in C2. 51st Annual Meeting of the Human Factors and Ergonomics Society, Baltimore, Oct. 1-5.
29.
RouxJ. N.van VuurenJ. H. (2007). Threat evaluation and weapon assignment decision support: A review of the state of the art. ORiON, 23, 151–186.
30.
SmallmanH. S.JohnM. S. (2003, October). CHEX (Change History EXplicit): New HCI concepts for change awareness. In Proceedings of the Human Factors and Ergonomics Society Annual Meeting (Vol. 47, No. 3, pp. 528-532). SAGE Publications.
31.
JohnM. S.SmallmanH. S. (2008). Staying up to speed: Four design principles for maintaining and recovering situation awareness. Journal of Cognitive Engineering and Decision Making, 2(2), 118-139.
32.
St JohnM.SmallmanH. S.ManesD. I. (2005). Assisted focus: Heuristic automation for guiding users’ attention toward critical information. In Proc. of the 1st inter. conf. on augmented cognition.
33.
VachonF.VallièresB. R.JonesD. M.TremblayS. (2012). Nonexplicit change detection in complex dynamic settings: What eye movements reveal. Human Factors, 54, 996–1007.
34.
VachonF.LafondD.VallièresB.R.RousseauR.TremblayS. (2011). Supporting situation awareness: A tradeoff between benefits and overhead. IEEE Conference on Cognitive Methods in Situation Awareness and Decision Support (pp. 284-291). Miami, FL, Feb. 22-24.
