Data visualizations typically show a representation of a data set with little to no focus on the repeatability or generalizability of the displayed trends and patterns. However, insights gleaned from these visualizations are often used as the basis for decisions about future events. Visualizations of retrospective data therefore often serve as “visual predictive models.” However, this visual predictive model approach can lead to invalid inferences. In this article, we describe an approach to visual model validation called Inline Replication. Inline Replication is closely related to the statistical techniques of bootstrap sampling and cross-validation and, like those methods, provides a non-parametric and broadly applicable technique for assessing the variance of findings from visualizations. This article describes the overall Inline Replication process and outlines how it can be integrated into both traditional and emerging “big data” visualization pipelines. It also provides examples of how Inline Replication can be integrated into common visualization techniques such as bar charts and linear regression lines. Results from an empirical evaluation of the technique and two prototype Inline Replication–based visual analysis systems are also described. The empirical evaluation demonstrates the impact of Inline Replication under different conditions, showing that both (1) the level of partitioning and (2) the approach to aggregation have a major influence over its behavior. The results highlight the trade-offs in choosing Inline Replication parameters but suggest that using partitions is a reasonable default.
ShneidermanB. The eyes have it: a task by data type taxonomy for information visualizations. In: Proceedings of IEEE symposium on visual languages, Boulder, CO, 3–6 September 1996, pp. 336–343. New York: IEEE.
2.
CardSMackinlayJShneidermanB.Readings in information visualization: using vision to think. San Francisco, CA: Morgan Kaufmann Publishers, 1999.
3.
ChiEH. A taxonomy of visualization techniques using the data state reference model. In: IEEE symposium on information visualization, Salt Lake City, UT, 9–10 October 2000, pp. 69–75. New York: IEEE.
4.
OstleB.Statistics in research. 2nd ed.Ames, IA: The Iowa State University Press, 1963.
GotzDStavropoulosH.DecisionFlow: visual analytics for high-dimensional temporal event sequence data. IEEE T Vis Comput Gr2014; 20: 1783–1792.
8.
SkeelsMLeeBSmithG, et al. Revealing uncertainty for information visualization. Inform Visual2010; 9(1): 70–81.
9.
PotterKRosenPJohnsonCR.From quantification to visualization: a taxonomy of uncertainty visualization approaches. IFIP Adv Inf Commun Technol2012; 377: 226–249.
10.
MacEachrenARothRO’BrienJ, et al. Visual semiotics and uncertainty visualization: an empirical study. IEEE T Vis Comput Gr2012; 18(12): 2496–2505.
11.
SanyalJZhangSBhattacharyaG, et al. A user study to compare four uncertainty visualization methods for 1D and 2D datasets. IEEE T Vis Comput Gr2009; 15(6): 1209–1218.
12.
TakSToetAvan ErpJ.The perception of visual uncertainty representation by non-experts. IEEE T Vis Comput Gr2013; 20: 935–943.
13.
PotterKGerberSAndersonEW.Visualization of uncertainty without a mean. IEEE Comput Graph2013; 33(1): 75–79.
14.
BrodlieKOsorioRALopesA. A review of uncertainty in data visualization. In: DillJEarnshawRKasikD, et al. (eds) Expanding the frontiers of visual analytics and visualization. Berlin: Springer, 2012, pp. 81–109.
15.
BorlandDWangWGotzD.Contextual visualization: making the unseen visible to combat bias during visual analysis. IEEE Comput Graph2018; 38(6): 17–23.
16.
GotzDSunSCaoN.Adaptive contextualization: combating bias during high-dimensional visualization and data selection. In: Proceedings of the 21st international conference on intelligent user interfaces, Sonoma, CA, 7–10 March 2016, pp. 85–95. New York: ACM.
17.
GotzDSunSCaoN, et al. Adaptive contextualization methods for combating selection bias during high-dimensional visualization. ACM T Interact Intell Syst2017; 7(4): 17-1–17-23.
18.
van den ElzenSvan WijkJ. BaobabView: interactive construction and analysis of decision trees. In: IEEE conference on visual analytics science and technology (VAST), Providence, RI, 23–28 October 2011, pp. 151–160. New York: IEEE.
19.
MuhlbacherTPiringerH.A partition-based framework for building and validating regression models. IEEE T Vis Comput Gr2013; 19(12): 1962–1971.
20.
GosinkLBensemaKPulsipherT, et al. Characterizing and visualizing predictive uncertainty in numerical ensembles through bayesian model averaging. IEEE T Vis Comput Gr2013; 19(12): 2703–2712.
21.
SipesNSMartinMTReifDM, et al. Predictive models of prenatal developmental toxicity from ToxCast high-throughput screening data. Toxicol Sci2011; 124: 109–127.
StolperCPererAGotzD.Progressive visual analytics: user-driven visual exploration of in-progress analytics. IEEE T Vis Comput Gr2014; 20(12): 1653–1662.
24.
FisherDPopovIDruckerS, et al. Trust me, I’m partially right: incremental visualization lets analysts explore large datasets faster. In: Proceedings of the SIGCHI conference on human factors in computing systems (CHI ’12), Austin, TX, 5–10 May 2012, pp. 1673–1682. New York: ACM.
25.
SheskinDJ.Handbook of parametric and nonparametric statistical procedures. 3rd ed.Boca Raton, FL: CRC Press, 2003.
26.
StolineMR.The status of multiple comparisons: simultaneous estimation of all pairwise comparisons in one-way ANOVA designs. Am Stat1981; 35(3): 134–141.
27.
RothmanKJ.No adjustments are needed for multiple comparisons. Epidemiology1990; 1(1): 43–46.
28.
ChenHZhangSChenW, et al. Uncertainty-aware multidimensional ensemble data visualization and exploration. IEEE T Vis Comput Gr2015; 21: 1072–1086.
29.
FengDKwockLLeeY, et al. Matching visual saliency to confidence in plots of uncertain data. IEEE T Vis Comput Gr2010; 16(6): 980–989.
30.
WuYYuanGXMaKL.Visualizing flow of uncertainty through analytical processes. IEEE T Vis Comput Gr2012; 18(12): 2526–2535.
31.
MajumderMHofmannHCookD.Validation of visual statistical inference, applied to linear models. J Am Stat Assoc2013; 108(503): 942–956.
32.
WickhamHCookDHofmannH, et al. Graphical inference for infovis. IEEE T Vis Comput Gr2010; 16(6): 973–979.
33.
CorreaCChanYHMaKL. A framework for uncertainty-aware visual analytics. In: IEEE symposium on visual analytics science and technology, Atlantic City, NJ, 12–13 October 2009, pp. 51–58. New York: IEEE.
34.
MunznerT.Visualization analysis and design. HAR/PSC ed. Boca Raton, FL: A K Peters/CRC Press, 2014.
SutcliffeABruijnOThewS, et al. Developing visualization-based decision support tools for epidemiology. Inform Visual2014; 13(1): 3–17.
39.
ZgraggenEZhaoZZeleznikR, et al. Investigating the effect of the multiple comparisons problem in visual analysis. In: Proceedings of the CHI conference on human factors in computing systems, Montreal, QC, Canada, 21–26 April 2018. New York: ACM.
40.
KohaviR. A study of cross-validation and bootstrap for accuracy estimation and model selection. In: Proceedings of the 14th international joint conference on artificial intelligence (IJCAI’95), Montreal, QC, Canada, 20–25 August 1995, vol. 2, pp. 1137–1143. San Francisco, CA: Morgan Kaufmann Publishers.
41.
EfronB.Bootstrap methods: another look at the jackknife. Ann Stat1979; 7(1): 1–26.
42.
LamLSuenSY.Application of majority voting to pattern recognition: an analysis of its behavior and performance. IEEE T Syst Man Cy A1997; 27(5): 553–568.
43.
GoldbergerALAmaralLANGlassL, et al. PhysioBank, PhysioToolkit, and PhysioNet: components of a new research resource for complex physiologic signals. Circulation2000; 101(23): e215–e220.
44.
GoodmanSN.Toward evidence-based medical statistics. 1: the P value fallacy. Ann Intern Med1999; 130(12): 995–1004.
