The Cognitive Shadow is a decision-support system that uses policy capturing to model human operators’ judgment policies and provide online predictions of their decisions. The system can provide support in reaction to a decision mismatch (shadowing mode) or proactively (recommendation mode). The goal of this study was to compare these two modes of operation in their ability to effectively model and support decision-making and to examine impacts on information processing, workload, and trust. Participants took part in an aircraft threat evaluation simulation without decision support or with the Cognitive Shadow (either shadowing or recommendation mode). Dwell time was collected over different areas of the user interface. While the recommendation mode had no advantage over the control group, the shadowing mode resulted in greater human and model accuracy. This mode led to longer dwell time over the parameters zone presenting key information for decision-making. These benefits were maintained even after the tool was removed. Workload was unaffected by the mode, and while trust was initially higher in the recommendation mode, it quickly became equivalent between both modes, overall supporting shadowing as the better configuration for cognitive assistance. Results are discussed in terms of decision processes, operators support, and automation bias.
ArmstrongJ. S. (2001). Judgmental bootstrapping: Inferring experts’ rules for forecasting. In ArmstrongJ. S. (Ed.), Principles of forecasting. International series in operations research & management science (vol. 30, pp. 171–192). Springer. https://doi.org/10.1007/978-0-306-47630-3_9
2.
ArsintescuL.ChachadR.GregoryK. B.MulliganJ. F.Flynn-EvansE. E. (2020). The relationship between workload, performance and fatigue in a short-haul airline. Chronobiology International, 37(9–10), 1492–1494. https://doi.org/10.1080/07420528.2020.1804924
3.
BacklundL. G.BringJ.SkånérY.StrenderL. E.MontgomeryH. (2009). Improving fast and frugal modeling in relation to regression analysis: Test of 3 models for medical decision making. Medical Decision Making, 29(1), 140–148. https://doi.org/10.1177/0272989X08326091
4.
BahnerJ. E.HüperA. D.ManzeyD. (2008). Misuse of automated decision aids: Complacency, automation bias and the impact of training experience. International Journal of Human-Computer Studies, 66(9), 688–699. https://doi.org/10.1016/j.ijhcs.2008.06.001
BalfeN.SharplesS.WilsonJ. R. (2015). Impact of automation: Measurement of performance, workload and behaviour in a complex control environment. Applied Ergonomics, 47, 52–64. https://doi.org/10.1016/j.apergo.2014.08.002
BowersC.DeatonJ.OserR.PrinceC.KolbM. (1995). Impact of automation on aircrew communication and decision-making performance. The International Journal of Aviation Psychology, 5(2), 145–167. https://doi.org/10.1207/s15327108ijap0502_2
9.
CamererC. (1981). General conditions for the success of bootstrapping models. Organizational Behavior and Human Performance, 27(3), 411–422. https://doi.org/10.1016/0030-5073(81)90031-3
10.
ChamberlandC.HodgettsH. M.VallièresB. R.VachonF.TremblayS. (2016). Pip and pop when auditory alarms facilitate visual change detection in dynamic settings. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 60(1), 284–288. https://doi.org/10.1177/1541931213601065
11.
ChatelaisB.LafondD.HainsA.GagnéC. (2020). Improving policy-capturing with active learning for real-time decision support. In AhramT.KarwowskiW.VerganoA.LealiF.TaiarR. (Eds.), Intelligent human systems integration 2020 (vol. 1131, pp. 177–182). Springer Nature.
12.
ClarkR. E.FeldonD. F.Van MerriënboerJ. J. G.YatesK. A.EarlyS. (2007). Cognitive task analysis. In SpectorJ. M.MerrillM. D.van MerriënboerJ. J. G.DriscollM. P. (Eds.), Handbook of research on educational communications and technology (3rd ed., pp. 577–594). Lawrence Erlbaum Associates.
13.
CookseyR. W. (1996). Judgment analysis: Theory, methods, and applications. Academic Press.
14.
CrandallB.KleinG. A. (2006). Working minds: A practitioner’s guide to cognitive task analysis. MIT Press.
15.
DekkerS. W. A.WoodsD. D. (1999). Automation and its impact on human cognition. In DekkerS. W. A.HollnagelE. (Eds.), Coping with computers in the cockpit (pp. 7–28). Ashgate Publishing Co.
16.
DuncanJ.WardR.ShapiroK. (1994). Direct measurement of attentional dwell time in human vision. Nature, 369(6478), 313–315. https://doi.org/10.1038/369313a0
17.
EdwardsT.GabetsC.MercerJ.BienertN. (2017). Task demande variation in air traffic control: Implications for workload, fatigue, and performance. In StantonN.LandryS.Di BucchianicoG.VallicelliA. (Eds.), Advances in human aspects of transportation. Advances in intelligent systems and computing (vol. 484, pp. 91–102). Springer.
18.
EndsleyM. R. (1987). The application of human factors to the development of expert systems for advanced cockpits. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 31(12), 1388–1392. https://doi.org/10.1177/154193128703101219
19.
EndsleyM. R.KirisE. O. (1995). The out-of-the-loop performance problem and level of control in automation. Human Factors, 37(2), 381–394. https://doi.org/10.1518/001872095779064555
20.
EstesS.HellebergJ.LongK.MenzenskiJ.MylesC.PollackM.QuezadaM.SteinJ. (2018). Principles for minimizing cognitive assistance distraction in the cockpit. In: 2018 IEEE/AIAA 37th digital avionics systems conference (pp. 23–27). DASC.
21.
EvansJ. S. B. T. (1988). The knowledge elicitation problem: a psychological perspective. Behaviour & Information Technology, 7(2), 111–130. https://doi.org/10.1080/01449298808901868
22.
FanJ.SmithA. P. (2017). The impact of workload and fatigue on performance. In LongoL.LevaM. (Eds.), Human mental workload: Models and applications. H-Workload 2017 (vol. 726, pp. 90–105). Springer.
23.
FordJ. K.SchmittN.SchechtmanS. L.HultsB. M.DohertyM. L. (1989). Process tracing methods: Contributions, problems, and neglected research questions. Organizational Behavior and Human Decision Processes, 43(1), 75–117. https://doi.org/10.1016/0749-5978(89)90059-9
24.
FunkK.LyallB.WilsonJ.VintR.NiemczykM.SuroteguhC.OwenG. (1999). Flight deck automation issues. The International Journal of Aviation Psychology, 9(2), 109–123. https://doi.org/10.1207/s15327108ijap0902_2
25.
GalarM.FernandezA.BarrenecheaE.BustinceH.HerreraF. (2012). A review on ensembles for the class imbalance problem: Bagging-boosting-and hybrid-based approaches. IEEE Transactions on Systems, Man, and Cybernetics—Part C: Applications and Reviews, 42(4), 463–484. https://doi.org/10.1109/TSMCC.2011.2161285
26.
GaoM.ChangR.WangA.ZhangM.ChengZ.LiQ.TangX. (2021). Which can explain the pip-and-pop effect during a visual search: Multisensory integration or the oddball effect?Journal of Experimental Psychology: Human Perception and Performance, 47(5), 689–703. https://doi.org/10.1037/xhp0000905
27.
GoddardK.RoudsariA.WyattJ. C. (2012). Automation bias: a systematic review of frequency, effect mediators, and mitigators. Journal of American Medical Informatics Association, 19(1), 121–127. https://doi.org/10.1136/amiajnl-2011-000089
28.
GrossetêteL.MaroisA.LafondD.ChatelaisB.GagnéC. (2022). Active learning for capturing human decision policies in a data frugal context. In NicosiaG.OjhaV.La MalfaE.La MalfaG.JansenG.PardalosP. M.GiuffridaG.UmetonR. (Eds.), Machine learning, optimization, and data science. LOD 2021. Lecture notes in computer science (pp. 395–407). Springer International Publishing.
HartS. G.StavelandL. E. (1988). Development of NASA-TLX (Task Load Index): Results of empirical and theoretical research. Advances in Psychology, 52, 139–183. https://doi.org/10.1016/S0166-4115(08)62386-9
31.
HodgettsH. M.TremblayS.VallièresB. R.VachonF. (2015). Decision support and vulnerability to interruption in a dynamic multitasking environment. International Journal of Human-Computer Studies, 79, 106–117. https://doi.org/10.1016/j.ijhcs.2015.01.009
HughesR. W.HurlstoneM. J.MarshJ. E.VachonF.JonesD. M. (2013). Cognitive control of auditory distraction: Impact of task difficulty, foreknowledge, and working memory capacity support duplex- mechanism account. Journal of Experimental Psychology: Human Perception and Performance, 39(2), 539–553. https://doi.org/10.1037/a0029064
34.
HughesR. W.VachonF.JonesD. M. (2007). Disruption of short-term memory by changing and deviant sounds: Support for a duplex-mechanism account of auditory distraction. Journal of Experimental Psychology: Learning, Memory, and Cognition, 33(6), 1050–1061. https://doi.org/10.1037/0278-7393.33.6.1050
35.
JanssenC. P.DonkerS. F.BrumbyD. P.KunA. L. (2019). History and future of human-automation interaction. International Journal of Human-Computer Studies, 131, 99–107. https://doi.org/10.1016/j.ijhcs.2019.05.006
36.
JianJ. Y.BisantzA. M.DruryC. G. (2000). Foundations for an empirically determined scale of trust in automated systems. International Journal of Cognitive Ergonomics, 4(1), 53–71. https://doi.org/10.1207/S15327566IJCE0401_04
37.
KarelaiaN.HogarthR. M. (2008). Determinants of linear judgment: A meta-analysis of lens model studies. Psychological Bulletin, 134(3), 404–426. https://doi.org/10.1037/0033-2909.134.3.404
38.
KaufmannE.WittmannW. W. (2016). The success of linear bootstrapping models: Decision domain-expertise-and criterion-specific meta-analysis. Plos One, 11(6), e0157914. https://doi.org/10.1371/journal.pone.0157914
39.
KawamotoK.HoulihanC. A.BalasE. A.LobachD. F. (2005). Improving clinical practice using clinical decision support systems: A systematic review of trials to identify features critical to success. Bmj: British Medical Journal, 330(7494), 765. https://doi.org/10.1136/bmj.38398.500764.8F
40.
KirkR. E. (1995). Experimental design: Procedures for the behavioral sciences (2nd ed.). Brooks/Cole Publishing Company.
41.
LabontéK.LafondD.ChatelaisB.HunterA.AkpanF.NeyedliH. F.TremblayS. (2021). Combining process tracing and policy capturing for judgment analysis in an anti-submarine warfare simulation. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 65(1), 1557–1561. https://doi.org/10.1177/1071181321651113
42.
LabontéK.LafondD.HunterA.NeyedliH. F.TremblayS. (2020). Comparing two decision support modes using the Cognitive Shadow online policy-capturing system. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 64(1), 1125–1129. https://doi.org/10.1177/1071181320641270
43.
LabontéK.TremblayS.VachonF. (2019). Forewarning interruptions in dynamic settings: Can prevention bolster recovery?Journal of Experimental Psychology: Applied, 25(4), 674–694. https://doi.org/10.1037/xap0000219
44.
LafondD.ChampagneJ.HervetG.GagnonJ.-F.TremblayS.RousseauR. (2009). Decision analysis using policy capturing and process tracing techniques in a simulated naval air-defense task. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 53(18), 1220–1224. https://doi.org/10.1177/154193120905301812
45.
LafondD.LabontéK.HunterA.NeyedliH. F.TremblayS. (2020). Judgment analysis for real-time decision support using the cognitive shadow policy-capturing system. In AhramT.TaiarR.ColsonS.ChoplinA. (Eds.), Human interaction and emerging Technologies. IHIET 2019. Advances in intelligent systems and computing, (vol. 1018, pp. 78–83). Springer.
46.
LafondD.VallièresB. R.VachonF.TremblayS. (2017). Judgment analysis in a dynamic multi-task environment: Capturing non-linear policies using decision trees. Journal of Cognitive Engineering and Decision Making, 11(2), 122–135. https://doi.org/10.1177/1555343416661889
47.
LaurikkalaJ. (2001). Improving identification of difficult small classes by balancing class distribution. In Proceedings of the 8th Conference On AI in Medicine, (pp. 63–66). Springer.
48.
LawB.AtkinsM. S.KirkpatrickA. E.LomaxA. J.MackenzieC. L. (2004). Eye gaze patterns differentiate novice and experts in a virtual laparoscopic surgery training environment. In Proceedings of the 2004 symposium on eye tracking research & applications. ETRA (pp. 41–48).
49.
MarshJ. E.CampbellT. A.VachonF.TaylorP.HughesR. W. (2020). How the deployment of visual attention modulates auditory distraction. Attention, Perception, & Psychophysics, 82(1), 350–362. https://doi.org/10.3758/s13414-019-01800-w
50.
MarshJ. E.SörqvistP.HughesR. W. (2015). Dynamic cognitive control of irrelevant sound: Increased task engagement attenuates semantic auditory distraction. Journal of Experimental Psychology: Human Perception & Performance, 41(5), 1462–1474. https://doi.org/10.1037/xhp0000060
51.
MorrisonJ. G.KellyR. T.MooreR. A.HutchinsS. G. (1998). Implications of decision-making research for decision support and displays. In Cannon-BowersJ. A.SalasE. (Eds.), Making decisions under stress: Implications for individual and team training (pp. 375–406). American Psychological Association.
52.
MosierK. L.SkitkaL. J.HeersS.BurdickM. (1998). Automation bias: Decision making and performance in high-tech cockpits. The International Journal of Aviation Psychology, 8(1), 47–63. https://doi.org/10.1207/s15327108ijap0801_3
53.
NeyedliH. F.HollandsJ. G.JamiesonG. A. (2011). Beyond identity: Incorporating system reliability information into an automated combat identification system. Human Factors, 53(4), 338–355. https://doi.org/10.1177/0018720811413767
54.
NokesK.HodgkinsonG. P. (2018). Policy-capturing: An ingenious technique for exploring the cognitive bases of work-related decisions. In GalavanR. J.SundK. J.HodgkinsonG. P. (Eds.), Methodological challenges and advances in managerial and organizational cognition (pp. 95–121). Emerald Publishing Limited.
55.
O’ConnorM.RemusW.LinK. (2005). Improving judgmental forecasts with judgmental bootstrapping and task feedback support. Journal of Behavioral Decision Making, 18(4), 247–260. https://doi.org/10.1002/bdm.499
56.
ParasuramanR. (2000). Designing automation for human use: empirical studies and quantitative models. Ergonomics, 43(7), 931–951. https://doi.org/10.1080/001401300409125
57.
ParasuramanR.ManzeyD. H. (2010). Complacency and bias in human use of automation: An attentional integration. Human Factors, 52(3), 381–410. https://doi.org/10.1177/0018720810376055
58.
ParmentierF. B. R. (2008). Towards a cognitive model of distraction by auditory novelty: the role of involuntary attention capture and semantic processing. Cognition, 109(3), 345–362. https://doi.org/10.1016/j.cognition.2008.09.005
59.
PashlerH.CepedaJ.WixtedJ. T.RohrerD. (2005). When does feedback facilitate learning of words?Journal of Experimental Psychology: Learning, Memory, and Cognition, 31(1), 3–8. https://doi.org/10.1037/0278-7393.31.1.3
60.
PeysakhovichV.VachonF.VallièresB. R.DehaisF.TremblayS. (2015). Pupil dilation and eye movements can reveal upcoming choice in dynamic decision-making. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 59(1), 210–214. https://doi.org/10.1177/1541931215591043
61.
PooleA.BallL. J. (2006). Eye tracking in human-computer interaction and usability research. Current status and future prospects. In GhaouiC. (Ed.), Encyclopedia of human computer interaction (pp. 211–219). Idea Group.
62.
RasmussenJ. (1983). Skills, rules, and knowledge; Signals, signs, and symbols, and other distinctions in human performance models. IEEE Transactions on Systems, Man, and Cybernetics, 13(3), 257–266. https://doi.org/10.1109/TSMC.1983.6313160
SchriverA. T.MorrowD. G.WickensC. D.TalleurD. A. (2008). Expertise differences in attentional strategies related to pilot decision making. Human Factors, 50(6), 864–878. https://doi.org/10.1518/001872008X374974
65.
Schulte-MecklenbeckM.JohnsonJ. G.BöckenholtU.GoldsteinD. G.RussoJ. E.SullivanN. J.WillemsenM. C. (2017). Process-tracing methods in decision making: On growing up in the 70s. Current Directions in Psychological Science, 26(5), 442–450. https://doi.org/10.1177/0963721417708229
66.
ShiffB. (1983). An experimental study of the human-computer interface in process control. University of Toronto, Department of Industrial Engineering. Unpublished Thesis.
67.
SkitkaL. J.MosierK. L.BurdickM. (1999). Does automation bias decision-making?International Journal of Human-Computer Studies, 51(5), 991–1006. https://doi.org/10.1006/ijhc.1999.0252
68.
SodergrenM. H.Orihuela-EspinaF.MountneyP.ClarkJ.TeareJ.DarziA.YangG.-Z. (2011). Orientation strategies in natural orifice transluminal endoscopic surgery. Annals of Surgery, 254(2), 257–266. https://doi.org/10.1097/SLA.0b013e31822513c6
69.
StaszewskiJ. J. (2008). Cognitive engineering based on expert skill: Notes on success and surprises. In SchraagenJ. M.MilitelloL. G.OrmerodT.LipshitzR. (Eds.), Naturalistic decision making and macrocognition (pp. 317–347). Ashgate.
70.
StrauchB. (2018). Ironies of automation: Still unresolved after all these years. IEEE Transactions on Human-Machine Systems, 48(5), 419–433. https://doi.org/10.1109/THMS.2017.2732506
71.
TienT.PucherP. H.SodergrenM. H.SriskandarajahK.YangG.-Z.DarziA. (2014). Eye tracking for skills assessment and training: A systematic review. Journal of Surgical Research, 191(1), 169–178. https://doi.org/10.1016/j.jss.2014.04.032
72.
VachonF.HughesR. W.JonesD. M. (2012). Broken expectations: Violation of expectancies, not novelty, captures auditory attention. Journal of Experimental Psychology: Learning, Memory, and Cognition, 38(1), 164–177. https://doi.org/10.1037/a0025054
73.
VachonF.LabontéK.MarshJ. E. (2017). Attentional capture by deviant sounds: A general form of distraction?Journal of Experimental Psychology: Learning, Memory, and Cognition, 43(4), 622–634. https://doi.org/10.1037/xlm0000330
74.
Van de MerweK.van DijkH.ZonR. (2012). Eye movements as an indicator of situation awareness in a flight simulator experiment. The International Journal of Aviation Psychology, 22(1), 78–95. https://doi.org/10.1080/10508414.2012.635129
75.
Van der BurgE.JohnC.OliversC. N.JanT.DavidA. (2010). Efficient visual search from synchronized auditory signals requires transient audiovisual events. Plos One, 5(5), e10664. https://doi.org/10.1371/journal.pone.0010664
76.
Van der BurgE.OliversC. N.BronkhorstA. W.TheeuwesJ. (2008). Pip and pop: Nonspatial auditory signals improve spatial visual search. Journal of Experimental Psychology: Human Perception and Performance, 34(5), 1053–1065. https://doi.org/10.1037/0096-1523.34.5.1053
77.
VansteenkisteP.CardonG.PhilippaertsR.LenoirM. (2014). Measuring dwell time percentage from head-mounted eye-tracking data – comparison of a frame-by-frame and a fixation-by-fixation analysis. Ergonomics, 58(5), 1–10. https://doi.org/10.1080/00140139.2014.990524
78.
VasilevM. R.ParmentierF. B. R.AngeleB.KirkbyJ. A. (2019). Distraction by deviant sounds during reading: An eye-movement study. The Quarterly Journal of Experimental Psychology, 72(7), 1863–1875. https://doi.org/10.1177/1747021818820816
79.
VõM. L.-H.AizenmanA. M.WolfeJ. M. (2016). You think you know where you looked? You better look again. Journal of Experimental Psychology: Human Perception and Performance, 42(10), 1477–1481. https://doi.org/10.1037/xhp0000264
80.
WickensC.TsangP. S. (2015). Workload. In Boehm-DavisD. A.DursoF. T.LeeJ. D. (Eds.), APA handbook of human systems integration (pp. 277–292). American Psychological Association.
WolpertD. H.MacreadyW. G. (1996). No free lunch theorems for optimization. IEEE Transactions on Evolutionary Computation, 1(1), 67–82. https://doi.org/10.1109/4235.585893
83.
WongS. S. H.LimS. W. H. (2019). Prevention-permission-promotion: A review of approaches to errors in learning. Educational Psychologist, 54(1), 1–19. https://doi.org/10.1080/00461520.2018.1501693
84.
YanZ.KantolaR.ZhangP. (2011). A research model for human-computer trust interaction. In 2011 IEEE 10th International Conference on Trust, Security and Privacy in Computing and Communications, (pp. 274–281). IEEE.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
