Comparing Two Decision Support Modes Using the Cognitive Shadow Online Policy-Capturing System

Abstract

The Cognitive Shadow is a prototype tool intended to support decision making by autonomously modeling human operators’ response pattern and providing online notifications to the operators about the decision they are expected to make in new situations. Since the system can be configured either in a reactive “shadowing” or a proactive “recommendation” mode, this study aimed to determine its most effective mode in terms of human and model accuracy, workload, and trust. Subjects participated in an aircraft threat evaluation simulation without decision support or while using either mode of the Cognitive Shadow. Whereas the recommendation mode had no advantage over the control condition, the shadowing mode led to higher human and model accuracy. These benefits were maintained even when the tool was unexpectedly removed. Neither mode influenced workload, and the initial lower trust rating in the shadowing mode faded quickly, making it the best overall configuration for the cognitive assistant.

Get full access to this article

View all access options for this article.

References

Backlund

L. G.

Bring

Skånér

Strender

L. E.

Montgomery

(2009). Improving fast and frugal modeling in relation to regression analysis: Test of 3 models for medical decision making. Med. Decis. Making, 29, 140–148.

Bahner

J. E.

Hüper

A. D.

Manzey

(2008). Misuse of automated decision aids: Complacency, automation bias and the impact of training experience. Int. J. Hum. Comput. Stud., 66, 688–699.

Balfe

Sharples

Wilson

J. R.

(2015). Impact of automation: Measurement of performance, workload and behaviour in a complex control environment. Appl. Ergon., 47, 52–64.

Berry

D. C.

(1987). The problem of implicit knowledge. Expert Syst., 4, 144–151.

Chatelais

Lafond

Hains

Gagné

(2020). Improving policy- capturing with active learning for real-time decision support. Proceedings of the Int. Conference on Intelligent Human Systems Integration, 1131, 177–182.

Cooksey

R. W.

(1996). Judgment analysis: Theory, methods, and applications. San Diego, CA: Academic Press.

Endsley

M. R.

Kiris

E. O.

(1995). The out-of-the-loop performance problem and level of control in automation. Hum. Factors, 37, 381–394.

Evans

J. S. B. T

. (1988). The knowledge elicitation problem: a psychological perspective. Behav. Inform. Technol., 7, 111–130.

Hart

S. G.

Staveland

L. E.

(1988). Development of NASA-TLX (Task Load Index): Results of empirical and theoretical research. Advances in Psychology, 52, 139–183.

10.

Jian

J. Y.

Bisantz

A. M.

Drury

C. G.

(2000). Foundations for an empirically determined scale of trust in automated systems. Int. J. Cognit. Ergon., 4, 53–71.

11.

Karelaia

Hogarth

R. M.

(2008). Determinants of linear judgment: A meta-analysis of lens model studies. Psychol. Bull., 134, 404–426.

12.

Lafond

Labonté

Hunter

Neyedli

H. F.

Tremblay

(2019). Judgment analysis for real-time decision support using the Cognitive Shadow policy-capturing system. Proceedings of the Int. Conference on Human Interaction and Emerging Technologies, 1018, 78–83.

13.

Lafond

Vallières

B. R.

Vachon

Tremblay

(2017). Judgment analysis in a dynamic multi-task environment: Capturing non-linear policies using decision trees. J. Cogn. Eng. Decis. Mak., 11, 122–135.

14.

Morrison

J. G.

Kelly

R. T.

Moore

R. A.

Hutchins

S. G.

(1998). Implications of decision-making research for decision support and displays. In Cannon-Bowers

J. A.

Salas

(Eds.), Making decisions under stress: Implications for individual and team training (pp. 375–406). Washington, DC: American Psychological Association.

15.

Neyedli

H. F.

Hollands

J. G.

Jamieson

G. A.

(2011). Beyond identity: Incorporating system reliability information into an automated combat identification system. Hum. Factors, 53, 338–355.

16.

Wolpert

D. H.

(1996). The lack of a priori distinctions between learning algorithms. Neural Comput., 8, 1341–1390.