Going beyond the desktop to leverage novel devices—such as smartphones, tablets, or large displays—for visual sensemaking typically requires supporting extraneous operations for device discovery, interaction sharing, and view management. Such operations can be time-consuming and tedious and distract the user from the actual analysis. Embodied interaction models in these multi-device environments can take advantage of the natural interaction and physicality afforded by multimodal devices and help effectively carry out these operations in visual sensemaking. In this article, we present cross-device interaction models for visualization spaces, that are embodied in nature, by conducting a user study to elicit actions from participants that could trigger a portrayed effect of sharing visualizations (and therefore information) across devices. We then explore one common interaction style from this design elicitation called Visfer, a technique for effortlessly sharing visualizations across devices using the visual medium. More specifically, this technique involves taking pictures of visualizations, or rather the QR codes augmenting them, on a display using the built-in camera on a handheld device. Our contributions include a conceptual framework for cross-device interaction and the Visfer technique itself, as well as transformation guidelines to exploit the capabilities of each specific device and a web framework for encoding visualization components into animated QR codes, which capture multiple frames of QR codes to embed more information. Beyond this, we also present the results from a performance evaluation for the visual data transfer enabled by Visfer. We end the article by presenting the application examples of our Visfer framework.
JakobsenMRSahlemariam HaileYKnudsenSet al. Information visualization and proxemics: design opportunities and empirical findings. IEEE T Vis Comput Gr2013; 19(12): 2386–2395.
2.
McGrathWBowmanBMcCallumDet al. Branch-explore-merge: facilitating real-time revision control in collaborative visual exploration. In: Proceedings of the ACM conference on interactive tabletops and surfaces, Cambridge, MA, 11–14 November 2012, pp. 235–244. New York: ACM.
3.
RobertsJRitsosPBadamSKet al. Visualization beyond the desktop—the next big thing. IEEE Comput Graph2014; 34(6): 26–34.
4.
IsenbergPElmqvistNScholtzJet al. Collaborative visualization: definition, challenges, and research agenda. Inform Visual2011; 10(4): 310–326.
5.
ElmqvistNIraniP.Ubiquitous analytics: interacting with big data anywhere, anytime. Computer2013; 46(4): 86–89.
6.
BadamSKFisherEElmqvistN.Munin: a peer-to-peer middleware for ubiquitous analytics and visualization spaces. IEEE T Vis Comput Gr2015; 21(2): 215–228.
7.
DourishP.Where the action is: the foundations of embodied interaction. Cambridge, MA: MIT Press, 2004.
8.
BallRNorthCBowmanDA.Move to improve: promoting physical navigation to increase user performance with large displays. In: Proceedings of the ACM conference on human factors in computing systems, San Jose, CA, 28 April–3 May 2007, pp. 191–200. New York: ACM.
9.
BadamSKElmqvistN. PolyChrome: a cross-device framework for collaborative web visualization. In: Proceedings of the ACM conference on interactive tabletops and surfaces, Dresden, 16–19 November 2014, pp. 109–118. New York: ACM.
10.
MarrinanTAurisanoJNishimotoAet al. SAGE2: a new approach for data intensive collaboration using scalable resolution shared displays. In: IEEE international conference on collaborative computing: networking, applications and worksharing (CollaborateCom), Miami, FL, 22–25 October 2014, pp. 177–186. New York: IEEE.
11.
BostockMOgievetskyVHeerJ.D: data-driven documents. IEEE T Vis Comput Gr2011; 17(12): 2301–2309.
12.
YonezawaTOgawaMKyonoYet al. SENseTREAM: enhancing online live experience with sensor-federated video stream using animated two-dimensional code. In: Proceedings of the ACM international joint conference on pervasive and ubiquitous computing, Seattle, WA, 13–17 September 2014, pp. 301–305. New York: ACM.
13.
ChungHNorthCSelfJZet al. VisPorter: facilitating information sharing for collaborative sensemaking on multiple displays. Pers Ubiquit Comput2014; 18(5): 1169–1186.
14.
KimKJavedWWilliamsCet al. Hugin: a framework for awareness and coordination in mixed-presence collaborative information visualization. In: Proceedings of the ACM conference on interactive tabletops and surfaces, Saarbrücken, 7–10 November 2010, pp. 231–240. New York: ACM.
15.
MacKayWE.Is paper safer? The role of paper flight strips in air traffic control. ACM T Comput-Hum Int1999; 6(4): 311–340.
16.
IsenbergPHinrichsUHancockMet al. Information visualization on interactive tabletops in work vs. public settings. In: Proceedings: Collaborative Visualization on Interactive Surfaces - CoVIS ‘09: Technical Report LMU-MI-2010-2. Atlantic, US, 11 October 2009, pp. 28–31. University of Munich.
17.
HamiltonPWigdorDJ.Conductor: enabling and understanding cross-device interaction. In: Proceedings of the ACM conference on human factors in computing systems, Toronto, ON, Canada, 26 April–1 May 2014, pp. 2773–2782. New York: ACM.
18.
RekimotoJ. Pick-and-drop: a direct manipulation technique for multiple computer environments. In: Proceedings of the ACM symposium on user interface software and technology, Banff, AB, Canada, 14–17 October 1997, pp. 31–39. New York: ACM.
19.
HinckleyKRamosGGuimbretiereFet al. Stitching: pen gestures that span multiple displays. In: Proceedings of the ACM conference on advanced visual interfaces, Gallipoli, 25–28 May 2004, pp. 23–31. New York: ACM.
20.
ChenXGrossmanTWigdorDJet al. Duet: exploring joint interactions on a smart phone and a smart watch. In: Proceedings of the ACM conference on human factors in computing systems, Toronto, ON, Canada, 26 April–1 May 2014, pp. 159–168. New York: ACM.
21.
HoubenSMarquardtN. WatchConnect: a toolkit for prototyping smartwatch-centric cross-device applications. In: Proceedings of the ACM conference on human factors in computing systems, Seoul, Republic of Korea, 18–23 April 2015, pp. 1247–1256. New York: ACM.
22.
von ZadowUBüschelWLangnerRet al. SleeD: using a sleeve display to interact with touch-sensitive display walls. In: Proceedings of the ACM conference on interactive tabletops and surfaces, Dresden, 16–19 November 2014, pp. 129–138. New York: ACM.
23.
YangJWigdorD. Panelrama: enabling easy specification of cross-device web applications. In: Proceedings of the ACM conference on human factors in computing systems, Toronto, ON, Canada, 26 April–1 May 2014, pp 2783–2792. New York: ACM.
24.
CzerwinskiMSmithGReganTet al. Toward characterizing the productivity benefits of very large displays. INTERACT’03, August 2003, vol. 3, pp. 9–16. IOS Press.
25.
AndrewsCEndertANorthC. Space to think: large high-resolution displays for sensemaking. In: Proceedings of the ACM conference on human factors in computing systems, Atlanta, GA, 10–15 April 2010, pp. 55–64. New York: ACM.
26.
BradelLEndertAKochKet al. Large high resolution displays for co-located collaborative sensemaking: display usage and territoriality. Int J Hum-Comput St2013; 71(11): 1078–1088.
27.
IsenbergPCarpendaleS.Interactive tree comparison for co-located collaborative information visualization. IEEE T Vis Comput Gr2007; 13(6): 1232–1239.
28.
IsenbergPFisherD.Collaborative brushing and linking for co-located visual analytics of document collections. Computer Graph Forum2009; 28: 1031–1038.
29.
KimKElmqvistN.Embodied lenses for collaborative visual queries on tabletop displays. Inform Visual2012; 11: 319–338.
30.
SpindlerMTominskiCSchumannHet al. Tangible views for information visualization. In: Proceedings of the ACM international conference on interactive tabletops and surfaces, Saarbrücken, 7–10 November 2010, pp. 157–166. New York: ACM.
31.
HallET.The hidden dimension. Garden City, NY: Anchor Books, 1966.
32.
BallendatTMarquardtNGreenbergS. Proxemic interaction: designing for a proximity and orientation-aware environment. In: Proceedings of the ACM conference on interactive tabletops and surfaces, Saarbrücken, 7–10 November 2010, pp. 121–130. New York: ACM.
33.
BadamSKAminiFElmqvistNet al. Supporting visual exploration for multiple users in large display environments. In: IEEE conference on visual analytics science and technology (VAST), Baltimore, MD, 23–28 October 2016.
34.
KisterUReipschlägerPMatulicFet al. BodyLenses: embodied magic lenses and personal territories for wall displays. In: Proceedings of the ACM international conference on interactive tabletops and surfaces, Madeira, 15–18 November 2015, pp. 117–126. New York: ACM.
35.
EndertAAndrewsCLeeYHet al. Visual encodings that support physical navigation on large displays. In: Proceedings of graphics interface, St. John’s, NL, Canada, 25–27 May 2011, pp. 103–110. Waterloo, ON, Canada: Canadian Human-Computer Communications Society.
36.
AndrewsCEndertAYostBet al. Information visualization on large, high-resolution displays: issues, challenges, and opportunities. Inform Visual2011; 10(4): 341–355.
37.
BallRDellaNoceMNiTet al. Applying embodied interaction and usability engineering to visualization on large displays. In: ACM British HCI - workshop on visualization & interaction, September 2006, pp. 57–65. New York: ACM.
38.
BallRNorthC.Realizing embodied interaction for visual analytics through large displays. Comput Graph2007; 31(3): 380–400.
39.
AndrewsCNorthC. Analyst’s workspace: an embodied sensemaking environment for large, high-resolution displays. In: Proceedings of the IEEE conference on visual analytics science and technology, Seattle, WA, 14–19 October 2012 pp. 123–131. New York: IEEE.
40.
JacobRJKGirouardAHirshfieldLMet al. Reality-based interaction: a framework for post-WIMP interfaces. In: Proceedings of the ACM conference on human factors in computing systems, Florence, 5–10 April 2008, pp. 201–210. New York: ACM.
41.
RaskarRWelchGCuttsMet al. The office of the future: a unified approach to image-based modeling and spatially immersive displays. In: Proceedings of the ACM conference on computer graphics and interactive techniques, Orlando, Florida, USA, 19–24 July 1998, pp. 179–188. New York: ACM.
42.
RaskarRBrownMSYangRet al. Multi-projector displays using camera-based registration. In: Proceedings of the IEEE conference on visualization, San Francisco, CA, 24 October 1999, pp. 161–522. Los Alamitos, CA: IEEE Computer Society Press.
43.
ScottDSharpRMadhavapeddyAet al. Using visual tags to bypass Bluetooth device discovery. ACM SIGMOBILE Mob Comput Commun Rev2005; 9(1): 41–53.
44.
KaltenbrunnerMBencinaR. ReacTIVision: a computer-vision framework for table-based tangible interaction. In: Proceedings of the ACM international conference on tangible and embedded interaction, Baton Rouge, LA, 15–17 February 2007, pp. 69–74. New York: ACM.
45.
KlokmoseCNKristensenJBBaggeRet al. BullsEye: high-precision fiducial tracking for table-based tangible interaction. In: Proceedings of the ACM international conference on interactive tabletops and surfaces, Dresden, 16–19 November 2014, pp. 269–278. New York: ACM.
46.
RädleRJetterHCMarquardtNet al. HuddleLamp: spatially-aware mobile displays for ad-hoc around-the-table collaboration. In: Proceedings of the ACM conference on interactive tabletops and surfaces, Dresden, 16–19 November 2014, pp. 45–54. New York: ACM.
47.
RohsMZweifelP. A conceptual framework for camera phone-based interaction techniques. In: International conference on pervasive computing, Munich, 8–13 May 2005, pp. 171–189. Berlin: Springer.
48.
HesselmannTHenzeNBollS.Flashlight: optical communication between mobile phones and interactive tabletops. In: Proceedings of the ACM international conference on interactive tabletops and surfaces, Saarbrücken, 7–10 November 2010, pp. 135–138. New York: ACM.
49.
LanglotzTBimberO.Unsynchronized 4D barcodes. In: BebisGBoyleRParvinBet al. (eds) Advances in visual computing. Berlin: Springer, 2007, pp. 363–374.
LiTAnCXiaoXet al. Real-time screen-camera communication behind any scene. In: Proceedings of the ACM conference on mobile systems, applications, and services, Florence, 18–22 May 2015, pp. 197–211. New York: ACM.
52.
TangAToryMPoBet al. Collaborative coupling over tabletop displays. In: Proceedings of the ACM conference on human factors in computing systems, Montréal, QC, Canada, 22–27 April 2006, pp. 1181–1190. New York: ACM.
53.
MorrisMRRyallKShenCet al. Beyond “social protocols”: multi-user coordination policies for co-located groupware. In: Proceedings of the ACM conference on computer supported cooperative work, Chicago, IL, 6–10 November 2004, pp. 262–265. New York: ACM.
54.
RedaKJohnsonAEPapkaMEet al. Effects of display size and resolution on user behavior and insight acquisition in visual exploration. In: Proceedings of the ACM conference on human factors in computing systems, Seoul, Republic of Korea, 18–23 April 2015, pp. 2759–2768. New York: ACM.
55.
ShoemakerGTsukitaniTKitamuraYet al. Body-centric interaction techniques for very large wall displays. In: Proceedings of the ACM Nordic conference on human-computer interaction: extending boundaries, Reykjavik, 16–20 October 2010, pp. 463–472. New York: ACM.
56.
ElmqvistNMoereAVJetterHCet al. Fluid interaction for information visualization. Inform Visual2011; 10(4): 327–340.
57.
LangnerRHorakTDachseltR. Towards combining mobile devices for visual data exploration. In: Poster program of the 2016 IEEE conference on information visualization (Infovis), Baltimore, Maryland, USA, 23–28 October 2016. New York: IEEE.
58.
AbrahamsonDTrninicD. Toward an embodied-interaction design framework for mathematical concepts. In: Proceedings of the ACM conference on interaction design and children, Ann Arbor, MI, 20–23 June 2011, pp. 1–10. New York: ACM.
59.
MarquardtNGreenbergS.Proxemic interactions: from theory to practice. San Rafael, CA: Morgan & Claypool Publishers, 2015.
60.
WobbrockJOMorrisMRWilsonAD. User-defined gestures for surface computing. In: Proceedings of the ACM SIGCHI conference on human factors in computing systems, Boston, MA, 4–9 April 2009, pp. 1083–1092. New York: ACM.
61.
RohsM.Marker-based embodied interaction for handheld augmented reality games. J Virtual Real Broadcast2007; 4(5): 1860–2037.
62.
AmatriainXKuchera-MorinJHollererTet al. The AlloSphere: immersive multimedia for scientific discovery and artistic exploration. IEEE Multimedia2009; 16(2): 64–75.
63.
TheveninDCoutazJ. Plasticity of user interfaces: framework and research agenda. In: Proceedings of INTERACT 99, Edinburgh, 30 August -3 September 1999, Vol. 99, pp. 110–117. IOS Press.
64.
HeerJShneidermanB.A taxonomy of tools that support the fluent and flexible use of visualizations. ACM Queue2012; 10(2): 1–26.
65.
ShneidermanB. The eyes have it: a task by data type taxonomy for information visualizations. In: Proceedings of the IEEE symposium on visual languages, Boulder, CO, 3–6 September 1996, pp. 336–343. New York: IEEE.
66.
YiJSKangYStaskoJTet al. Toward a deeper understanding of the role of interaction in information visualization. IEEE T Vis Comput Gr2007; 13(6): 1224–1231.
67.
SatyanarayanAWongsuphasawatKHeerJ. Declarative interaction design for data visualization. In: Proceedings of the ACM symposium on user interface software and technology, Honolulu, HI, 5–8 October 2014, pp. 669–678. New York: ACM.
68.
TobiaszMIsenbergPCarpendaleS.Lark: Coordinating co-located collaboration with information visualization. IEEE T Vis Comput Gr2009; 15(6): 1065–1072.
69.
PerlinKFoxD. Pad: an alternative approach to the computer interface. In: SIGGRAPH ’93: proceedings of the 20th annual conference on computer graphics and interactive techniques, Anaheim, CA, 2–6 August 1993, pp. 57–64. New York: ACM.
70.
IsenbergPDragicevicPWillettWet al. Hybrid-image visualization for large viewing environments. IEEE T Vis Comput Gr2013; 19(12): 2346–2355.
71.
WickerSBBhargavaVK.Reed-Solomon codes and their applications
. Hoboken, NJ: John Wiley & Sons, 1999.
72.
ChenWYangGZhangG. A simple and efficient image pre-processing for QR decoder. In: Proceedings of the international conference on electronic & mechanical engineering and information technology, Shenyang, Liaoning, China, 7–9 September 2012.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
