There are many challenging decision-making situations due to high degrees of uncertainty, turbulence, information burden, interdependencies and competing decision criteria in complex environments such as military command and control and emergency response. Unfortunately, few decision support systems help these cognitively challenged decision makers understand the relative desirability of one course of action versus another, also known as option awareness (OA). We hypothesized that providing novel data visualization displays to compare options across multiple criteria in parallel, combined with interaction mechanisms tailored to exploring options, would improve decision makers’ abilities to make better choices more confidently. To test our hypothesis, we created a design inspired by principles of OA and the forementioned challenges and instantiated it in a patented proof-of-concept implementation called the Multi-criteria Option Comparison Application (MOCA). We experimented with MOCA in command and control and emergency response scenarios with subject matter experts and obtained statistically significant quantitative data showing the participants believed their use of a MOCA-like OA application will result in more accurate and confident choices, an enhanced ability to consider multiple perspectives and relevant decision factors, and eight other benefits. Additionally, qualitative data analyses surfaced themes of innovation, discovery, collaboration, ambiguity reduction, assumption validation, and interactivity.
EndsleyMR. Situation awareness global assessment technique (SAGAT). In: Proceedings of the IEEE national aerospace and electronics conference. IEEE, Dayton, OH, 23–27 May 1988, pp.789–795.
2.
DruryJLKleinGLBookerL, et al, Reimagining situation awareness and option awareness for human-machine teaming. In: Proceedings of the IEEE conference on cognitive and computational aspects of situation management (CogSIMA). Salerno, Italy, 6–10 June 2022, pp.9–15. IEEE.
3.
DruryJLKleinGPfaffMS, et al. Dynamic decision support for emergency responders. In: IEEE conference on technologies for homeland security. Waltham, Massachusetts, 11–12 May 2009, pp.537–544. IEEE.
4.
PfaffMSKleinGLDruryJL, et al. Supporting complex decision making through option awareness. J Cogn Eng Decis Mak2013; 7: 155–178.
5.
PfaffMSDarlingEDruryJL, et al. Patient empowerment with shared decision spaces. In: Procedia technology: HCIST 2014—international conference on health and social care information systems and technologies (eds VarajãoJCunhaMBjørn-AndersenNet al.), Troia, Portugal, 15–17 October 2014, pp.1477–1486. HCIST.
6.
LiuYMoonSPPfaffMS, et al. Collaborative option awareness for emergency response decision making. In: Proceedings of the 8th International Information Systems for Crisis Response and Management (ISCRAM) Conference, 2011 May. Belgium: ISCRAM.
7.
JohnsonMBradshawJMFeltovichPJ, et al. Coactive design: designing support for interdependence in joint activity. J Hum Robot Interact2014; 3(1):43–69.
8.
ThompsonJD. Organizations in action: social science bases of administrative strategy. 1st ed.McGraw-Hill, 1967.
9.
KleinGLTracyBAHainesE, et al. Multi-criteria option comparison tool. US Patent #12,153,785 B2. November26, 2024.
10.
DruryJLPfaffMSKleinGL, et al. Decision space visualization: lessons learned and design principles. In: Human-computer interaction. interaction modalities and techniques: 15th international conference, HCI international, Las Vegas, NV, 21–26 July 2013, pp.658–667. Springer Berlin Heidelberg.
11.
LamH.A framework of interaction costs in information visualization. IEEE Trans Vis Comput Graph2008; 14(6):1149–1156.
12.
HullmanJAdarEShahP.Benefitting infovis with visual difficulties. IEEE Trans Vis Comput Graph2011; 17(12):2213–2222.
13.
NielsenJMolichR. Heuristic evaluation of user interfaces. In: Proceedings of the SIGCHI conference on human factors in computing systems, ACM, Seattle, WA, 1990 March 1, pp.249–256.
14.
HeerJShneidermanB.Interactive dynamics for visual analysis: A taxonomy of tools that support the fluent and flexible use of visualizations. Queue2012; 10(2):30–55.
15.
RheingansP.Are we there yet? Exploring with dynamic visualization. IEEE Comput Graph Appl2002; 22: 6–10.
16.
ShneidermanB.The future of interactive systems and the emergence of direct manipulation. Behav Inf Technol1982; 1(3): 237–256.
17.
BaldonadoMQWWoodruffAGKuchinskyA. Guidelines for using multiple views in information visualization. In: Proceedings of the working conference on advanced visual interfaces, ACM, Palermo, Italy, 1 May 2000, pp.110–119.
18.
LempertRJ.Robust decision making (RDM). In: MarchauVWalkerWBloemenPPopperS (eds) Decision making under deep uncertainty: from theory to practice. 1st ed.Springer, 2019, pp.23–51.
19.
SmithJEWinklerRL.The optimizer’s curse: skepticism and postdecision surprise in decision analysis. Manag Sci2006; 52: 311–322.
20.
ChandrasekaranB. Designing decision support systems to help avoid biases & make robust decisions, with examples from army planning. In: Proceedings of the Army Science Conference, DTIC, Orlando, FL, 1–4 December 2008.
21.
ChandrasekaranBGoldmanM. Exploring robustness of plans for simulation-based course of action planning: a framework and an example. In: Proceedings of the IEEE symposium on computational intelligence in multi-criteria decision-making, Honolulu, HI, 1–5 April 2007, pp.185–192. IEEE.
22.
KalraNHallegatteSLempertR, et al. Agreeing on robust decisions: new processes for decision making under deep uncertainty. World Bank Policy Research Working Paper. 2014 (6906).
23.
MatrosovESWoodsAMHarouJJ.Robust decision making and info-gap decision theory for water resource system planning. J Hydrol2013; 494: 43–58.
24.
Casal-CamposAFuGButlerD, et al. An integrated environmental assessment of green and gray infrastructure strategies for robust decision making. Environ Sci Technol2015; 49(14): 8307–8314.
25.
HadkaDHermanJReedP, et al. An open-source framework for many-objective robust decision making. Environ Modell Softw2015; 74: 114–129.
ComfortLKColellaBVoortmanM, et al. Real-time decision making in urgent events: Modeling options for action. In: Proceedings of the 10th International conference on information systems for crisis response and management (ISCRAM), ISCRAM, Baden-Baden, Germany, 2013 May.
28.
MathieuJPfaffMSDruryJL, et al. Tactical robust decision-making methodology: effect of disease spread model fidelity on option awareness. In: Proceedings of the 7th International Information Systems for Crisis Response and Management (ISCRAM) Conference (eds FrenchSTomaszewskiBMZobelCW), Seattle, Washington, 2–5 May 2010. Belgium: ISCRAM.
29.
PfaffMSDruryJLKleinGL.Crowdsourcing mental models using DESIM (Descriptive to Executable Simulation Modeling). In: Proceedings of the Naturalistic Decision Making (NDM) Conference, McLean, Virginia, 9–12 June 2015, pp.1–7. Virginia: NDM.
30.
DruryJLPfaffMSKleinGL.A method for rigorously assessing causal mental models to support distributed team cognition. In: McNeeseMSalasEEndsleyMR (eds), Handbook of distributed team cognition, Vol. 2, Contemporary research: Models, methodologies, and measures in team cognition. CRC Press, 2020, pp.115–135.
GalesicMGigerenzerGStraubingerN.Natural frequencies help older adults and people with low numeracy to evaluate medical screening tests. Med Decis Mak2009; 29(3): 368–371.
33.
Garcia-RetameroRHoffrageU.Visual representation of statistical information improves diagnostic inferences in doctors and their patients. Soc Sci Med2013; 83: 27–33.
34.
HoffrageUKraussSMartignonL, et al. Natural frequencies improve Bayesian reasoning in simple and complex inference tasks. Front Psychol2015; 6: 1473.
35.
TaitARVoepel-LewisTZikmund-FisherBJ, et al. The effect of format on parents’ understanding of the risks and benefits of clinical research: a comparison between text, tables, and graphs. J Health Commun2010; 15: 487–501.
36.
TurchioeMRGrossmanLVMyersAC, et al. Visual analogies, not graphs, increase patients’ comprehension of changes in their health status. J Am Med Inform Assoc2020; 27: 677–689.
37.
RoundtreeKAGoodrichMAAdamsJA.Transparency: transitioning from human-machine systems to human-swarm systems. J Cogn Eng Decis Mak2019; 13: 171–195.
38.
ChenJYLakhmaniSGStowersK, et al. Situation awareness-based agent transparency and human-autonomy teaming effectiveness. Theor Issues Ergonomics Sci2018; 19(3): 259–282.
39.
ChenJYProcciKBoyceM, et al. Situation awareness-based agent transparency. ARL-TR-6905. US Army Research Laboratory. 2014.
40.
BruniSMarquezJJBrzezinskiA, et al. Introducing a human-automation collaboration taxonomy (HACT) in command and control decision-support systems. In: 12th international command and control research and technology symposium. Newport, MA: Command & Control Research Program, 2007, pp.1–13.
41.
OsoskySSandersTJentschF, et al. Determinants of system transparency and its influence on trust in and reliance on unmanned robotic systems. In: Proceedings of SPIE Defense and Security (eds KarlsenREGageDWShoemakerCMGerhartGR), Baltimore, MD, USA, 5–9 May 2014, pp.1–12. Bellingham: SPIE.
42.
LyonsJBSadlerGGKoltaiK, et al. Shaping trust through transparent design: theoretical and experimental guidelines. In: Savage-KnapshieldPChenJYC (eds) Advances in human factors in robots and unmanned systems. 1st ed.Springer, 2016, pp.127–136.
43.
BankesS.Exploratory modeling for policy analysis. Oper Res1993; 41(3): 435–449.
44.
HeinrichJWeiskopfD.State of the art of parallel coordinates. In: SbertMSzirmay-KalosL (eds) Eurographics 2013—state of the art reports. The Eurographics Association, 2013.
45.
InselbergA.The plane with parallel coordinates. Vis Comput1985; 1: 69–91.
46.
SiirtolaHLaivoTHeimonenT, et al. Visual perception of parallel coordinate visualizations. In: Proceedings of the 13th International Conference on Information Visualisation, Barcelona, Spain, 15–17 July 2009, pp.3–9. IEEE.
47.
LiZ.Interactive edge-bundled parallel coordinates. Master’s Thesis, University of Victoria, Canada, 2021.
48.
KuangXZhangHZhaoS, et al. Tracing tuples across dimensions: a comparison of scatterplots and parallel coordinate plots. Comput Graph Forum2012; 31(3): 1365–1374.
49.
NetzelRVuongJEngelkeU, et al. Comparative eye-tracking evaluation of scatterplots and parallel coordinates. Vis Inform2017; 1: 118–131.
50.
PalmasGBachynskyiMOulasvirtaA, et al. An edge-bundling layout for interactive parallel coordinates. In: Proceedings of the 2014 IEEE pacific visualization symposium, Yokohama, Japan, 4–7 March 2014, pp.57–64. IEEE.
51.
KleinGA.Sources of power: how people make decisions. MIT Press, 1998.
52.
PhillipsJKKleinGASieckWR.Expertise in judgment and decision making: a case for training intuitive decision skills. In: KoehlerDJHarveyN (eds) Blackwell handbook of judgment and decision making. 1st ed.Malden: Blackwell Publishing, Ltd., 2004, pp.297–315.
53.
WilliamsonCShneidermanB. The dynamic HomeFinder: evaluating dynamic queries in a real-estate information exploration system. In: Proceedings of the 15th annual international ACM SIGIR conference on research and development in information retrieval (eds BelkinNIngwersenPPejtersenAM), Copenhagen, Denmark, 21–24 June 1992, pp.338–346. SIGIR.
54.
GlaserBStraussA.Discovery of grounded theory: Strategies for qualitative research. Aldine Publishing Co., 1967.
55.
HeathC.Crisis management and multimedia technology in London underground line control rooms. J Comput Supp Coop Work1992; 1(1): 24–48.
56.
SimonHAEricssonKA.Protocol analysis: verbal reports as data. MIT Press, 1984.
57.
RicherGSansenJLalanneF, et al. Enabling hierarchical exploration for large-scale multidimensional data with abstract parallel coordinates. In: Proceedings of international workshop on big data visual exploration and analytics (BigVis), Vienna, Austria, March 2018, pp.76–83. BigVis.
58.
WangJLiuXShenH, et al. Multi-resolution climate ensemble parameter analysis with nested parallel coordinates plots. IEEE Trans Vis Comput Graph2017; 23: 81–90.
59.
AissiHRoyB.Robustness in multi-criteria decision aiding. In: EhrgottMFigueiraJRGrecoS (eds) Trends in multiple criteria decision analysis. Springer, 2010, pp.87–121.
