Predicting and capturing an analyst’s intent behind a selection in a data visualization is valuable in two scenarios: First, a successful prediction of a pattern an analyst intended to select can be used to auto-complete a partial selection which, in turn, can improve the correctness of the selection. Second, knowing the intent behind a selection can be used to improve recall and reproducibility. In this paper, we introduce methods to infer analyst’s intents behind selections in data visualizations, such as scatterplots. We describe intents based on patterns in the data, and identify algorithms that can capture these patterns. Upon an interactive selection, we compare the selected items with the results of a large set of computed patterns, and use various ranking approaches to identify the best pattern for an analyst’s selection. We store annotations and the metadata to reconstruct a selection, such as the type of algorithm and its parameterization, in a provenance graph. We present a prototype system that implements these methods for tabular data and scatterplots. Analysts can select a prediction to auto-complete partial selections and to seamlessly log their intents. We discuss implications of our approach for reproducibility and reuse of analysis workflows. We evaluate our approach in a crowd-sourced study, where we show that auto-completing selection improves accuracy, and that we can accurately capture pattern-based intent.
BrehmerMMunznerT.A multi-level typology of abstract visualization tasks. IEEE Trans Vis Comput Graph2013; 19(12): 2376–2385.
4.
RindAAignerWWagnerM, et al. Task cube: a three-dimensional conceptual space of user tasks in visualization design and evaluation. Inf Vis2016; 15(4): 288–300.
5.
KarerBHagenHLehmannDJ.Insight beyond numbers: the impact of qualitative factors on visual data analysis. IEEE Trans Vis Comput Graph2021; 27(2): 1011–1021.
6.
MyersBA.Creating User Interfaces by Demonstration. PhD Thesis, University of Toronto, Canada, 1987.
NguyenPHXuKWheatA, et al. SensePath: understanding the sensemaking process through analytic provenance. IEEE Trans Vis Comput Graph2016; 22(1): 41–50.
9.
DimitriadouKPapaemmanouilODiaoY. Explore-by-example: an automatic query steering framework for interactive data exploration. In: Proceedings of the 2014 ACM SIGMOD international conference on management of data. SIGMOD’14, Snowbird, UT, 22–27 June 2014, pp.517–528. New York: ACM.
10.
CavalloMDemiralpC.Clustrophile 2: guided visual clustering analysis. IEEE Trans Vis Comput Graph2018; 25(1): 267–276.
11.
DouWJeongDHStukesF, et al. Recovering reasoning processes from user interactions. IEEE Comput Graph Appl2009; 29(3): 52–61.
12.
NorthCChangREndertA, et al. Analytic provenance: process+interaction+insight. In: CHI’11 extended abstracts on human factors in computing systems. CHI EA’11, Vancouver, BC, Canada, 7–12 May 2011, pp.33–36. New York: ACM.
13.
BrownETOttleyAZhaoH, et al. Finding Waldo: learning about users from their interactions. IEEE Trans Vis Comput Graph2014; 20(12): 1663–1672.
14.
CenedaDGschwandtnerTMikschS.A review of guidance approaches in visual data analysis: a multifocal perspective. Comput Graph Forum2019; 38(3): 861–879.
15.
OttleyAGarnettRWanR.Follow the clicks: learning and anticipating mouse interactions during exploratory data analysis. Comput Graph Forum2019; 38(3): 41–52.
16.
SteichenBCareniniGConatiC.User-adaptive information visualization: using eye gaze data to infer visualization tasks and user cognitive abilities. In: Proceedings of the 2013 international conference on intelligent user interfaces. IUI’13, Santa Monica, CA, 19–22 March 2013, pp.317–328. New York: ACM.
17.
GingerichMJConatiC. Constructing models of user and task characteristics from eye gaze data for user-adaptive information highlighting. In: Twenty-Ninth AAAI conference on artificial intelligence, Austin, Texas, 25 January 2015, pp.1728–1734. Menlo Park, CA: AAAI Press.
18.
MonadjemiSGarnettROttleyA.Competing models: inferring exploration patterns and information relevance via Bayesian model selection. IEEE Trans Vis Comput Graph2021; 27(2): 412–421.
19.
BattleLChangRStonebrakerM. Dynamic prefetching of data tiles for interactive visualization. In: Proceedings of the 2016 international conference on management of data. SIGMOD’16, San Francisco, California, 26 June–1 July 2016, pp.1363–1375. New York: ACM.
20.
MackinlayJHanrahanPStolteC.Show Me: automatic presentation for visual analysis. IEEE Trans Vis Comput Graph2007; 13(6): 1137–1144.
21.
ToryMSetlurV. Do what i mean, not what i say! design considerations for supporting intent and context in analytical conversation. In: 2019 IEEE conference on visual analytics science and technology (VAST), Vancouver, BC, Canada, 20–25 October 2019, pp.93–103. New York: IEEE.
22.
SaketBKimHBrownET, et al. Visualization by demonstration: an interaction paradigm for visual data exploration. IEEE Trans Vis Comput Graph2017; 23(1): 331–340.
23.
SaketBJiangLPerinC, et al. Liger: combining interaction paradigms for visual analysis. arXiv:190708345 [cs]2019; 1907.08345.
FanCHauserH.Fast and accurate CNN-based brushing in scatterplots. Comput Graph Forum2018; 37(3): 111–120.
26.
MartinARWardMO. High dimensional brushing for interactive exploration of multivariate data. In: Proceedings of the IEEE conference on visualization (Vis’95), Atlanta, GA, 29 October–3 November 1995, pp.271–278. Washington, DC: IEEE Computer Society Press.
27.
DerthickMKolojejchickJRothSF. An interactive visual query environment for exploring data. In: Proceedings of the 10th annual ACM symposium on user interface software and technology. UIST’97, Banff, AB, Canada, 14–17 October 1997, pp.189–198. New York: ACM.
28.
ShneidermanB.Dynamic queries for visual information seeking. IEEE Softw1994; 11(6): 70–77.
29.
HeerJAgrawalaMWillettW.Generalized selection via interactive query relaxation. In: Proceedings of the SIGCHI conference on human factors in computing systems. CHI’08, Florence, Italy, 5–10 April 2008, pp.959–968. New York: ACM.
30.
SuSLParisSDurandF.QuickSelect: history-based selection expansion. In: Proceedings of graphics interface 2009. GI’09, Kelowna, BC, Canada, 25–27 May 2009, pp.215–221. Toronto, ON: Canadian Information Processing Society.
31.
XiaoLGerthJHanrahanP. Enhancing visual analysis of network traffic using a knowledge representation. In: 2006 IEEE symposium on visual analytics science and technology, Baltimore, MD, 31 October–2 November 2006, pp.107–114. New York: IEEE.
32.
RaganEDEndertASanyalJ, et al. Characterizing provenance in visualization and data analysis: an organizational framework of provenance types and purposes. IEEE Trans Vis Comput Graph2016; 22(1): 31–40.
33.
FreireJKoopDSantosE, et al. Provenance for computational tasks: a survey. Comput Sci Eng2008; 10(3): 11–21.
34.
DeelmanEGannonDShieldsM, et al. Workflows and e-science: an overview of workflow system features and capabilities. Future Gener Comput Syst2009; 25(5): 528–540.
35.
ParkerSGJohnsonCR. SCIRun: a scientific programming environment for computational steering. In: Proceedings of the ACM/IEEE conference on supercomputing (SC’95), San Diego, CA, 8 December 1995, p.52. New York: ACM.
36.
BavoilLCallahanSPScheideggerC, et al. VisTrails: enabling interactive multiple-view visualizations. In: Proceedings of the IEEE conference on visualization (VIS’05), Minneapolis, MN, 23–28 October 2005, pp.135–142. New York: IEEE.
37.
DunneCHenry RicheNLeeB, et al. GraphTrail: analyzing large multivariate, heterogeneous networks while supporting exploration history. In: Proceedings of the SIGCHI conference on human factors in computing systems (CHI’12), Austin,Texas, 5–10 May 2012, pp.1663–1672. New York: ACM.
38.
YuBSilvaCT.VisFlow – web-based visualization framework for tabular data with a subset flow model. IEEE Trans Vis Comput Graph2017; 23(1): 251–260.
39.
CutlerZTGadhaveKLexA. Trrack: a library for provenance tracking in web-based visualizations. In: IEEE visualization conference (VIS), Salt Lake City, UT, 25–30 October 2020, pp.116–120. New York: IEEE.
40.
HeerJMackinlayJStolteC, et al. Graphical histories for visualization: supporting analysis, communication, and evaluation. IEEE Trans Vis Comput Graph2008; 14(6): 1189–1196.
41.
KadivarNChenVDunsmuirD, et al. Capturing and supporting the analysis process. In: 2009 IEEE symposium on visual analytics science and technology, Atlantic City, NJ, 12–13 October 2009, pp.131–138. New York: IEEE.
42.
KreuselerMNockeTSchumannH. A history mechanism for visual data mining. In: Proceedings of the IEEE symposium on information visualization (InfoVis’04), Austin, Texas, 10–12 October 2004, pp.49–56. New York: IEEE.
43.
ShrinivasanYBvan WijkJJ. Supporting the analytical reasoning process in information visualization. In: Proceedings of the SIGCHI conference on human factors in computing systems. CHI’08, Florence, Italy, 5–10 April 2008, pp.1237–1246. New York: ACM.
44.
StreitMSchulzHJLexA, et al. Model-driven design for the visual analysis of heterogeneous data. IEEE Trans Vis Comput Graph2012; 18(6): 998–1010.
45.
GratzlSLexAGehlenborgN, et al. From visual exploration to storytelling and back again. Comput Graph Forum2016; 35(3): 491–500.
46.
FujiwaraTCrnovrsaninTMaKL.Concise provenance of interactive network analysis. Vis Inform2018; 2: 213–224.
47.
StitzHGratzlSPiringerH, et al. KnowledgePearls: provenance-based visualization retrieval. IEEE Trans Vis Comput Graph2018; 25(1): 120–130.
48.
CamisettyAChandurkarCSunM, et al. Enhancing web-based analytics applications through provenance. IEEE Trans Vis Comput Graph2018; 25(1): 131–141.
49.
GrothDPStreefkerkK.Provenance and annotation for visual exploration systems. IEEE Trans Vis Comput Graph2006; 12(6): 1500–1510.
50.
WrightWSchrohDProulxP, et al. The sandbox for analysis: concepts and methods. In: Proceedings of the SIGCHI conference on human factors in computing systems, Montréal, Québec, 22–27 April 2006, pp.801–810. New York: ACM.
51.
WohlfartMHauserH. Story telling for presentation in volume visualization. In: Proceedings of the symposium on visualization (EuroVis’07), Norrköping, Sweden, 23–25 May 2007, pp.91–98. Goslar, Germany: Eurographics Association.
52.
EcclesRKaplerTHarperR, et al. Stories in GeoTime. Inf Vis2008; 7(1): 3–17.
53.
KwonBCStoffelFJäckleD, et al. VisJockey: enriching data stories through orchestrated interactive visualization. In: Poster compendium of the computation + journalism symposium, New York, NY, 24–25October2014. New York, NY: The Brown Institute for Media Innovation, Pulitzer Hall, Columbia University.
54.
GotzDWenZ.Behavior-driven visualization recommendation. In: Proceedings of the conference on intelligent user interfaces (IUI’09), Sanibel Island, Florida, 8–11 February 2009, pp.315–324. New York: ACM.
55.
AmarREaganJStaskoJ. Low-level components of analytic activity in information visualization. In: Proceedings of the IEEE symposium on information visualization (InfoVis’05), Minneapolis, MN, 23–25 October 2005, pp.111–117. New York: IEEE.
56.
SarikayaAGleicherM.Scatterplots: tasks, data, and designs. IEEE Trans Vis Comput Graph2018; 24(1): 402–412.
57.
BorzsonySKossmannDStockerK. The skyline operator. In: Proceedings 17th international conference on data engineering, Heidelberg, Germany, 2–6 April 2001, pp.421–430. New York: IEEE.
58.
PedregosaFVaroquauxGGramfortA, et al. Scikit-learn: machine learning in python. J Mach Learn Res2011; 12: 2825–2830.
59.
GratzlSLexAGehlenborgN, et al. LineUp: visual analysis of multi-attribute rankings. IEEE Trans Vis Comput Graph2013; 19(12): 2277–2286.
60.
CarpendaleS.Evaluating information visualizations. In: StaskoJTFeketeJDNorthC, et al. (eds) Information visualization: human-centered issues and perspectives. Lecture Notes in Computer Science. Berlin, Heidelberg: Springer, 2008, pp.19–45.
61.
WangZRitchieJZhouJ, et al. Data comics for reporting controlled user studies in human-computer interaction. IEEE Trans Vis Comput Graph2021; 27(2): 967–977.
62.
NobreCWoottonDHarrisonL, et al. Evaluating multivariate network visualization techniques using a validated design and crowdsourcing approach. In: Proceedings of the 2020 CHI conference on human factors in computing systems, Honolulu, HI, 25–30 April 2020, pp.1–12. New York: IEEE.
63.
DragicevicP.Fair statistical communication in HCI. In: RobertsonJKapteinM (eds) Modern statistical methods for HCI. Cham: Springer International Publishing, 2016, pp.291–330.
64.
CummingG. Understanding the new statistics: effect sizes, confidence intervals, and meta-analysis. Oxfordshire, England: Routledge, 2013.
65.
NobreCWoottonDCutlerZ, et al. reVISit: looking under the hood of interactive visualization studies. In: SIGCHI conference on human factors in computing systems (CHI) (to appear), Yokohama, Japan, 8–13 May 2021, pp.1–12. New York: ACM.
66.
BorgoRLeeBBachB, et al. Crowdsourcing for information visualization: promises and pitfalls. In: ArchambaultDPurchaseHHoßfeldT (eds) Evaluation in the crowd. Crowdsourcing and human-centered experiments. Cham: Springer, 2017, pp.96–138, Vol. 10264.
67.
FurmanovaKGratzlSStitzH, et al. Taggle: combining overview and details in tabular data visualizations. Inf Vis2020; 19(2): 114–136.
68.
LiuYKaleAAlthoffT, et al. Boba: authoring and visualizing multiverse analyses. IEEE Trans Vis Comput Graph2021; 27(2): 1753–1763.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
