Combining Process Tracing and Policy Capturing Techniques for Judgment Analysis in an Anti-Submarine Warfare Simulation

Abstract

The Cognitive Shadow is a prototype decision support tool that can notify users when they deviate from their usual judgment pattern. Expert decision policies are learned automatically online while performing one’s task using a combination of machine learning algorithms. This study investigated whether combining this system with the use of a process tracing technique could improve its ability to model human decision policies. Participants played the role of anti-submarine warfare commanders and rated the likelihood of detecting a submarine in different ocean areas based on their environmental characteristics. In the process tracing condition, participants were asked to reveal only the information deemed necessary, and only that information was sent to the system for model training. In the control condition, all the available information was sent to the system with each decision. Results showed that process tracing data improved the model’s ability to predict human decisions compared to the control condition.

Get full access to this article

View all access options for this article.

References

Armstrong

J. S.

(2001). Judgmental bootstrapping: Inferring experts’ rules for forecasting. In Principles of Forecasting (pp. 171–192). Springer.

Backlund

L. G.

Bring

Skånér

Strender

L. E.

Montgomery

(2009). Improving fast and frugal modeling in relation to regression analysis: Test of 3 models for medical decision making. Medical Decision Making, 29, 140–148.

Berry

D. C.

(1987). The problem of implicit knowledge. Expert Systems, 4, 144–151.

Billings

R. S.

Marcus

S. A.

(1983). Measures of compensatory and non-compensatory models of decision behavior: Process tracing versus policy capturing. Organizational Behavior and Human Performance, 31, 331–352.

Boekaerts

(1999). Self-regulated learning: where we are today. International Journal of Educational Research, 31, 445–457.

Chatelais

Lafond

Hains

Gagné

(2020). Improving policy-capturing with active learning for real-time decision support. Proceedings of the International Conference on Intelligent Human Systems Integration, 3, 177–182.

Cook

G. J.

Swain

M. R.

(1993). A computerized approach to decision process tracing for decision support system design. Decision Sciences, 24, 931–952.

Cooksey

R. W.

(1996). Judgment analysis: Theory, methods, and applications. Academic Press.

Evans

J. S. B. T

. (1988). The knowledge elicitation problem: a psychological perspective. Behaviour & Information Technology, 7, 111–130.

10.

Guerin

(2010). Social facilitation. In Weiner

I. B.

Craighead

W. E.

(Eds.), The Corsini encyclopedia of psychology (pp. 1–2). John Wiley & Sons, Inc.

11.

Labonté

Lafond

Hunter

Neyedli

H. F.

Tremblay

(2020). Comparing two decision support modes using the Cognitive Shadow online policy-capturing system. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 64, 1125–1129.

12.

Lafond

Champagne

Hervet

Gagnon

J. F.

Tremblay

Rousseau

(2009). Decision analysis using policy capturing and process tracing techniques in a simulated naval air-defense task. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 53, 1220–1224.

13.

Lafond

Labonté

Hunter

Neyedli

H. F.

Tremblay

(2019). Judgment analysis for real-time decision support using the Cognitive Shadow policy-capturing system. Proceedings of the International Conference on Human Interaction and Emerging Technologies, 1, 78–83.

14.

Lafond

Lacouture

Cohen

A. L.

(2009). Decision tree models of categorization response times, choice proportions, and typicality judgments. Psychological Review, 116, 833–855.

15.

Lafond

Vallières

B. R.

Vachon

Tremblay

(2017). Comparing naval decision support technologies using decision models, process tracing and error analysis. Proceedings to the Human Factors and Ergonomics Society Annual Meeting, 61, 1178–1182.

16.

Naval Oceanographic Office (1999). Fleet oceanographic and acoustic reference manual (RP33). Stennis Space Center, MS.

17.

Schulte-Mecklenbeck

Johnson

J. G.

Böckenholt

Goldstein

D. G.

Russo

J. E.

Sullivan

N. J.

Willemsen

M. C.

(2017). Process-tracing methods in decision making: On growing up in the 70s. Current Directions in Psychological Science, 26, 442–450.