Restricted accessResearch articleFirst published online 2025-6
Influence of Presenting Uncertainty Information on the Evaluation of Watershed Plans by Users of an Automation-Assisted Environmental Decision Support System
An important aspect of human-computer interfaces used to support automation-aided decision-making is the communication of uncertainty. However, people may misinterpret or ignore uncertainty information. We investigated how evaluations of suggested conservation plans in an environmental decision support system were influenced by visualizations displayed with and without uncertainty. We also examined whether test-users can be utilized to assess decision support systems (e.g., when end-user testing is not feasible), and whether users’ statistical backgrounds influenced decision making. Levels of uncertainty were manipulated via a novel Asymmetric Uncertainty Metric. Seventy-seven participants (53 test-users, 24 end-users) evaluated conservation plans in two counterbalanced online sessions, one presented with uncertainty information and one presented without. Linear regression results indicated that evaluations made with uncertainty differed from those without (Λ = 0.663, p < .001), suggesting that the presentation of uncertainty information in decision support systems can modify decisions about conservation plans. End-users were more influenced by the inclusion of uncertainty information than test-users, (bend-users = 0.22–0.28, btest-users = 0.08–0.12) (p < .001), and individuals who used (p = .016) or had been trained to use (p < .001) statistical data were also more sensitive to the inclusion of uncertainty information (bwsb = 0.17–0.18) than those without (bwnsb = 0.07–0.11).
AlhadadS. S. J. (2018). Visualizing data to support judgement, inference, and decision making in learning analytics: Insights from cognitive psychology and visualization science. Journal of Learning Analytics, 5(2), 60–85. https://doi.org/10.18608/jla.2018.52.5
2.
AndreA. D.CutlerH. A. (1998). Displaying uncertainty in advanced navigation systems. Proceedings of the Human Factors and Ergonomics Society - Annual Meeting, 42(1), 31–35. https://doi.org/10.1177/154193129804200108
3.
ArnoldJ. G.SrinivasanR.MuttiahR. S.WilliamsJ. R. (1998). Large area hydrologic modeling and assessment part I: Model development. JAWRA Journal of the American Water Resources Association, 34(1), 73–89. https://doi.org/10.1111/j.1752-1688.1998.tb05961.x
4.
Babbar-SebensM.MukhopadhyayS.SinghV. B.PiemontiA. D. (2015). A web-based software tool for participatory optimization of conservation practices in watersheds. Environmental Modelling & Software, 69, 111–127. https://doi.org/10.1016/j.envsoft.2015.03.011
5.
BartlettM. L.McCarleyJ. S. (2017). Benchmarking aided decision making in a signal detection task signal detection task. Human Factors: The Journal of the Human Factors and Ergonomics Society, 59(6), 881–900. https://doi.org/10.1177/0018720817700258
6.
BeliaS.FidlerF.WilliamsJ.CummingG. (2005). Researchers misunderstand confidence intervals and standard error bars. Psychological Methods, 10(4), 389–396. https://doi.org/10.1037/1082-989X.10.4.389
7.
BooneA. P.GunalpP.HegartyM. (2018). Explicit versus actionable knowledge: The influence of explaining graphical conventions on interpretation of hurricane forecast visualizations. Journal of Experimental Psychology: Applied, 24(3), 275–295. https://doi.org/10.1037/xap0000166
ChavasJ.KristjansonP.MatlonP. (1991). On the role of information in decision making: The case of sorghum yield in Burkina Faso. Journal of Development Economics, 35(2), 261–280. https://doi.org/10.1016/0304-3878(91)90051-v
10.
CorrellM.GleicherM. (2014). Error bars considered harmful: Exploring alternate encodings for mean and error. IEEE Transactions on Visualization and Computer Graphics, 20(12), 2142–2151. https://doi.org/10.1109/TVCG.2014.2346298
11.
CovelloV. T.von WinterfeldtD.SlovicP. (1986). Risk communication: A review of the literature. Risk Abstracts, 3(4), 171–182.
12.
DessartF. J.Barreiro-HurléJ.Van BavelR. (2019). Behavioural factors affecting the adoption of sustainable farming practices: A policy-oriented review. European Review of Agricultural Economics, 46(3), 417–471. https://doi.org/10.1093/erae/jbz019
13.
EdwardsJ. A.SnyderF. J.AllenP. M.MakinsonK. A.HambyD. M. (2012). Decision making for risk management: A comparison of graphical methods for presenting quantitative uncertainty. Risk Analysis, 32(12), 2055–2070. https://doi.org/10.1111/j.1539-6924.2012.01839.x
14.
FioreS. M.SongJ.NewtonO. B.PittmanC.WartaS. F.LaViolaJ. J. (2019). Determining the effect of training on uncertainty visualization evaluations. In AhramT.FalcãoC. (Eds.), Advances in usability, user experience and assistive technology. AHFE 2018. Advances in intelligent systems and computing (Vol. 794, pp. 141–152). Springer. https://doi.org/10.1007/978-3-319-94947-5_14
15.
FranconeriS. L.PadillaL. M.ShahP.ZacksJ. M.HullmanJ. (2021). The science of visual data communication: What works. Psychological Science in the Public Interest, 22(3), 110–161. https://doi.org/10.1177/15291006211051956
16.
GroundsM. A.JoslynS. L. (2018). Communicating weather forecast uncertainty: Do individual differences matter?Journal of Experimental Psychology: Applied, 24(1), 18–33. https://doi.org/10.1037/xap0000165
17.
HaroldJ.LorenzoniI.ShipleyT. F.CoventryK. R. (2016). Cognitive and psychological science insights to improve climate change data visualization. Nature Climate Change, 6(12), 1080–1089. https://doi.org/10.1038/NCLIMATE3162
18.
HegartyM.FriedmanA.BooneA. P.BarrettT. J. (2016). Where are you? The effect of uncertainty and its visual representation on location judgments in GPS-like displays. Journal of Experimental Psychology: Applied, 22(4), 381–392. https://doi.org/10.1037/xap0000103
19.
HullmanJ.QiaoX.CorrellM.KaleA.KayM. (2019). In pursuit of error: A survey of uncertainty visualization evaluation. IEEE Transactions on Visualization and Computer Graphics, 25(1), 903–913. https://doi.org/10.1109/TVCG.2018.2864889
20.
IOM. (2013). Incorporating uncertainty into decision making & communication of uncertainty. In Environmental decisions in the face of uncertainty. Institute of Medicine of the National Academies. https://www.ncbi.nlm.nih.gov/books/NBK200849/
21.
IsendahlN.DewulfA.Pahl-WostlC. (2010). Making framing of uncertainty in water management practice explicit by using a participant-structured approach. Journal of Environmental Management, 91(4), 844–851. https://doi.org/10.1016/j.jenvman.2009.10.016
22.
JoslynS.LeClercJ. (2013). Decisions with uncertainty: The glass half full. Current Directions in Psychological Science, 22(4), 308–315. https://doi.org/10.1177/0963721413481473
JoslynS.SavelliS. (2021). Visualizing uncertainty for non-expert end users: The challenge of the deterministic construal error. Frontiers of Computer Science, 2(January), 1–12. https://doi.org/10.3389/fcomp.2020.590232
25.
JoslynS.SavelliS.Nadav-GreenbergL. (2011). Reducing probabilistic weather forecasts to the worst-case scenario: Anchoring effects. Journal of Experimental Psychology: Applied, 17(4), 342–353. https://doi.org/10.1037/a0025901
26.
KaleA.NguyenF.KayF.HullmanJ. (2019). Hypothetical outcome plots help untrained observers judge trends in ambiguous data. IEEE Transactions on Visualization and Computer Graphics, 25(1), 892–902. https://doi.org/10.1109/TVCG.2018.2864909
27.
KusumastutiS. A.PollardK. A.OiknineA. H.DalanginB.RaberT. R.FilesB. T. (2023). Practice improves performance of a 2D uncertainty integration task within and across visualizations. IEEE Transactions on Visualization and Computer Graphics, 29(9), 3949–3960. https://doi.org/10.1109/TVCG.2022.3173889
28.
LiuT.BruinsR. J. F.HeberlingM. T. (2018). Factors influencing farmers’ adoption of best management practices: A review and synthesis. EPA Public Access, 10(2), 432. https://doi.org/10.3390/su10020432
29.
McMahonR.StauffacherM.KnuttiR. (2015). The unseen uncertainties in climate change: Reviewing comprehension of an IPCC scenario graph. Climatic Change, 133(2), 141–154. https://doi.org/10.1007/s10584-015-1473-4
30.
MosierK. L.ManzeyD. (2020). Humans and automated decision aids: A match made in heaven? In MoulouaM.HancockP. A. (Eds.), Human performance in automated and autonomous systems: Current theory and methods (pp. 19–42). CRC Press.
31.
Nadav-GreenbergL.JoslynS. L. (2009). Uncertainty forecasts improve decision making among nonexperts. Journal of Cognitive Engineering and Decision Making, 3(3), 209–227. https://doi.org/10.1518/155534309X474460
NewmanG. E.SchollB. J. (2012). Bar graphs depicting averages are perceptually misinterpreted: The within-the-bar bias. Psychonomic Bulletin & Review, 19(4), 601–607. https://doi.org/10.3758/s13423-012-0247-5
34.
PadillaL. M.Creem-RegehrS. H.HegartyM.StefanucciJ. K. (2018). Decision making with visualizations: A cognitive framework across disciplines. Cognitive Research: Principles and Implications, 3(1), 29. https://doi.org/10.1186/s41235-018-0120-9
35.
PadillaL. M.RuginskiI. T.Creem-RegehrS. H. (2017). Effects of ensemble and summary displays on interpretations of geospatial uncertainty data. Cognitive Research: Principles and Implications, 2(1), 40. https://doi.org/10.1186/s41235-017-0076-1
PiemontiA. D.MacugaK. L.Babbar-SebensM. (2017). Usability evaluation of an interactive decision support system for user-guided design of scenarios of watershed conservation practices. Journal of Hydroinformatics, 19(5), 701–718. https://doi.org/10.2166/hydro.2017.017
38.
ReuzeauF. (2001). Finding the best users to involve in design: A rational approach. Dans Le Travail Humain, 64(3), 223–245. https://doi.org/10.3917/TH.643.0223
39.
RuginskiI. T.BooneA. P.PadillaL. M.LiuL.HeydariN.KramerH. S.HegartyM.ThompsonW. B.HouseD. H.Creem-RegehrS. H. (2016). Non-expert interpretations of hurricane forecast uncertainty visualizations. Spatial Cognition and Computation, 16(2), 154–172. https://doi.org/10.1080/13875868.2015.1137577
40.
ScownH.BartlettM.McCarleyJ. S. (2014). Statistically lay decision makers ignore error bars in two-point comparisons. Proceedings of the Human Factors and Ergonomics Society - Annual Meeting, 58(1), 1746–1750. https://doi.org/10.1177/1541931214581364
41.
SivaprasadS. (2017). What does decision making with intervals really assume? The relationship between the Hurwicz decision rule and prescriptive decision analysis. [Doctoral dissertation, [Iowa State University]. https://doi.org/10.31274/etd-180810-5845
42.
SongJ.NewtonO. B.FioreS. M.CoadJ.ClarkJ.PittmanC.LaViolaJ. J. (2018). Examining the impact of training and feedback on visualization-supported decision making under uncertainty. Proceedings of the 9th Annual Conference of Applied Human Factors and Ergonomics, 62(1), 1449–1453. https://doi.org/10.1177/1541931218621329
StricklandL.BoagR. J.HeathcoteA.BowdenV.LoftS. (2023). Automated decision aids: When are they advisors and when do they take control of human decision making?Journal of Experimental Psychology: Applied, 29(4), 849–868. https://doi.org/10.1037/xap0000463
45.
TakS.ToetA.Van ErpJ. (2014). The perception of visual uncertainty representation by non-experts. IEEE Transactions on Visualization and Computer Graphics, 20(6), 935–943. https://doi.org/10.1109/TVCG.2013.247
46.
TurnerM. M.SkubiszC.RimalR. N. (2011). Theory and practice in risk communication: A review of the literature and visions for the future. Routledge.
47.
WaldmanK. B.ToddP. M.OmarS.BlekkingJ. P.GirouxS. A.AttariS. Z.BaylisK.EvansT. P. (2020). Agricultural decision making and climate uncertainty in developing countries. Environmental Research Letters, 15(11), 113004. https://doi.org/10.1088/1748-9326/abb909
48.
WilbyR.MurphyC. (2019). Decision-making by water managers despite climate uncertainty. In PfefferT.SmithJ. B.EbiK. L. (Eds.), The oxford handbook of planning for climate change hazards. Oxford Academic. https://doi.org/10.1093/oxfordhb/9780190455811.013.52
49.
XuY. P.TungY. K. (2008). Decision-making in water management under uncertainty. Water Resources Management, 22(5), 535–550. https://doi.org/10.1007/s11269-007-9176-x
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
