Evaluating exploratory visualization systems: A user study on how clustering-based visualization systems support information seeking from large document collections

Abstract

Iterative, opportunistic and evolving visual sense-making has been an important research topic as it assists users in overcoming ever-increasing information overload. Exploratory visualization systems (EVSs) maximize the amount of information users can gain through learning and have been widely used in scientific discovery and decision-making contexts. Although many EVSs have been developed recently, there is a lack of general guidance on how to evaluate such systems. Researchers face challenges such as understanding the cognitive learning process supported by these systems. In this paper, we present a formal user study on Newdle, a clustering-based EVS for large news collections, shedding light on a general methodology for EVS evaluation. Our approach is built upon cognitive load theory, which takes the user as well as the system as the focus of evaluation. The carefully designed procedures allow us to thoroughly examine the user’s cognitive process as well as control the variability among human subjects. Through this study, we analyse how and why clustering-based EVSs benefit (or hinder) users in a variety of information-seeking tasks. We also summarize leverage points for designing clustering-based EVSs.

Keywords

Clustering cognition human–computer interaction information visualization information seeking

Get full access to this article

View all access options for this article.

References

Popova

. Data visualization: stories for the information age. BloomBerg Businessweek, 2009. http://www.businessweek.com/innovate/content/aug2009/id20090811_137179.htm.

White

Marchionini

Muresan

. Editorial: evaluating exploratory search systems. Inform Process Manag 2008; 44: 433–436.

Zamir

Etzioni

. Grouper: a dynamic clustering interface to web search results. Comput Netw 1999; 31: 1361–1374.

Wise

Thomas

Pennock

. Visualizing the non-visual: spatial analysis and interaction with information from text documents. In: Proceedings of IEEE information visualization, Atlanta, Georgia, USA: IEEE Computer Society, 1995, pp. 51–58.

Lee

Czerwinski

Robertson

. Understanding research trends in conferences using paperlens. In: Proceedings of extended abstracts of the ACM conference on human factors in computing systems, Portland, OR, USA: ACM, 2005, pp. 1969–1972.

Pirolli

Schank

Hearst

. Scatter/gather browsing communicates the topic structure of a very large text collection. In: Proceedings of the SIGCHI conference on human factors in computing systems, Vancouver, British Columbia, Canada: ACM, 1996, pp. 213–220.

Sweller

. Cognitive load during problem solving: effects on learning. Cognitive Sci 1988; 12: 257–285.

Shneiderman

. Designing the user interface: strategies for effective human–computer interaction. Addison-Wesley Longman Publishing Co. Inc., Boston, MA, USA, 2010.

Hearst

Elliott

English

. Finding the flow in web site search. Commun ACM 2002; 45: 42–49.

10.

Kules

Shneiderman

. Users can change their web search tactics: design guidelines for categorized overviews. Inf. Process. Manage 2008; 44: 463–484.

11.

Kammerer

Nairn

Pirolli

. Signpost from the masses: learning effects in an exploratory social tag search browser. In: Proceedings of the 27th international conference on human factors in computing systems, Boston, MA, USA: ACM, 2009, pp. 625–634.

12.

Hart

Stavenland

. Development of the NASA-TLX (task load index): results of empirical and theoretical research. In: Proceedings of human mental workload, Amsterdam: North Holland Press, 1988, pp. 139–183.

13.

Anderson

Potter

Matzen

. A user study of visualization effectiveness using EEG and cognitive load. Comput. Graph. Forum 2011; 30: 36–104.

14.

Kang

Stasko

. Lightweight task/application performance using single versus multiple monitors: a comparative study. In: Proceedings of graphics interface conference, Windsor, Ontario, Canada: Canadian Information Processing Society, 2008, pp. 17–24.

15.

Kellar

Watters

Shepherd

. A goal-based classification of web information tasks. In: Proceedings of ACM SIGCOMM, Austin, TX, USA: ACM, 2006, pp. 101–112.

16.

Käki

Aula

. Controlling the complexity in comparing search user interfaces via user studies. Inform Process Manag 2008; 44: 82–91.

17.

Carnegie Mellon University. Assessing Prior Knowledge, http://www.cmu.edu/teaching/designteach/teach/priorknowledge.html

18.

Johnson-Laird

. Mental models: towards a cognitive science of language, inference, and consciousness, Harvard University Press, Cambridge, MA, USA, 1983.

19.

Erdelez

Wang

F-K

. The effects of conceptual description and search practice on users’ mental models and information seeking in a case-based reasoning retrieval system. Inform Process Manag 2008; 44: 294–309.

20.

Brunken

Plass

Leutner

. Direct measurement of cognitive load in multimedia learning. Educ Psychol 2003; 38: 53–61.

21.

Paas

Tuovinen

Tabbers

. Cognitive load measurement as a means to advance cognitive load theory. Educ Psychol 2003; 38: 63–71.

22.

Yang

Luo

Liu

. Newdle: interactive visual exploration of large online news collections. IEEE Comput Graph 2010; 30: 32–41.

23.

Newman

. Finding community structure in networks using the eigenvectors of matrices. Phys Rev E 2006; 7: 36–104.

24.

Viegas

Wattenberg

Feinberg

. Participatory visualization with Wordle. Phys Rev E 2009; 15: 1134–1144.

25.

Veerasamy

Belkin

. Evaluation of a tool for visualization of information retrieval results. In: SIGIR 1996, Zurich, Switzerland: ACM, 1996, pp. 85–92.

26.

Green

Jeong

Fisher

. Using personality factors to predict interface learning performance. In: Proceedings of Hawaii international conference on system sciences, Hawaii, USA: IEEE Computer Society, 2010, pp. 1–10.