This paper reports on the findings of a user study which investigated the effects of structure, highlighting and quantity of information on university students’ interaction with metadata surrogates of learning object repositories (LORs) during the relevance judgement process. The literature review revealed a lack of studies investigating how metadata surrogates should be designed to meet users’ needs and improve the effectiveness of relevance judgement. In order to address this issue, different versions of a prototype system, called META-LOR 2, were developed and evaluated in terms of the time needed for users to find relevant information, the accuracy of their relevance judgements and their satisfaction. To evaluate the prototype, a user study was conducted where participants were asked to complete a set of tasks and fill in satisfaction questionnaires. The findings showed that participants performed better and were more satisfied with those versions of the prototype which highlighted metadata elements that included the query terms, used clusters or categories to organize metadata elements in the surrogates and included metadata elements which were relevant to the query or task at hand. The paper concludes with some recommendations for improving the design of metadata surrogates in search result interfaces.
M. Zeng, and J. Qin, Metadata (Facet, London, 2008).
2.
C. Barry, Document representations and clues to document relevance, Journal of the American Society for Information Science and Technology49 (1998) 1293-1303.
3.
G. Marchionini , Towards multimedia surrogates ( 2007). Available at: www.open-video.org/details.php?videoid=10220&PHPSESSID=834cffc91eb6651a2026f0f728bee1fd (accessed 1 March 2009).
4.
H. Joho and J. Jose, Effectiveness of additional representations for the search result presentation on the web, Journal of Information Processing and Management44 (2008) 226-241.
5.
O. DroriHow to display search results in digital libraries: user study. In: P. Isaias, F. Sedes, J. Augusto and U. Ultes-Nitsche (eds), Proceedings of the 3rd International Workshop on New Developments in Digital Library (NDDL 2003) ( ICEIS Press, 2003).
6.
R. Rele and A. Duchowski, Using eye tracking to evaluate alternative search results interfaces. In: Proceedings of the 49th Annual Meeting of Human Factors and Ergonomics Society (HFES, Orlando, FL, 2005).
7.
P. Balatsoukas , A. Morris and A. O’Brien, An evaluation framework of user interaction with metadata surrogates, Journal of Information Science35 (2009) 321-339.
8.
P. Balatsoukas , A. Morris and A. O’Brien, Designing metadata surrogates for search result interfaces of learning object repositories: linear versus clustered metadata design. In: L. Chan and B. Martens (eds), Proceedings of the 11th International Conference on Electronic Publishing ( ELPUB, Vienna, 2007).
9.
IeeeLtsc , Draft standard for learning object metadata ( 2002). Available at: http://ltsc.ieee.org/wg12/files/LOM_1484_12_1_v1_Final_Draft.pdf (accessed 18 June 2009).
10.
T. Saracevic , Relevance: a review of the literature and a framework for thinking on the notion in information science. Part II: Nature and manifestations of relevance, Journal of the American Society for Information Science and Technology58(13) 1915-1933.
11.
O. Drori , Improving display of search results in information retrieval systems - users’ study. Technical report no. 200034 (2000). Available at: www.global-report.com/drori/?l=he&a=3234 (accessed 10 October 2009).
12.
Y. Zhang and L. Yuelin, A user-centered functional metadata evaluation of moving image collections, Journal of the American Society for Information Science and Technology59(8) (2008) 1331-1346.
13.
J. Janes, Relevance judgments and the incremental presentation of document representations , Information Processing and Management27 (1991) 629-646.
14.
E. Liddy, E. Allen, C. Finneran, G. Gay, H. Hembrooke and L. Granka, MetaTest: evaluation of metadata from generation to use. In: Proceedings of the 3rd ACM/IEEE-CS Joint Conference on Digital Libraries (IEEE Computer Society, Washington, DC, 2003).
15.
C. Barry and L. Schamber, Users’ criteria for relevance evaluation: a cross-situational comparison , Information Processing and Management34 (1998) 219-236.
16.
P. Wang and D. Soergel, A cognitive model of document use during a research project. Study I. Document selection, Journal of the American Society for Information Science and Technology49 (1998 ) 115-133.
17.
A. Crystal and J. Greenberg, Relevance criteria identified by health information users during web searches , Journal of the American Society for Information Science and Technology57 (2006) 1368-2882.
18.
B. Fraser and M. Gluck, Usability of geospatial metadata or space-time matters, Bulletin of the American Society for Information Science, 25 (1999). Available at: www.asis.org/Bulletin/Aug-99/fraser_gluck.html(accessed 3 December 2008).
19.
Search LTevaluation report (2002). Available at: www.failte.ac.uk/documents/eval_report.rtf (accessed 12 July 2009).
20.
A. Diekema, Evaluating metadata from different perspectives (2008). Available at: www.ils.unc.edu/mrc/jcdl2006/slides/diekema.pdf (accessed 6 June 2008).
21.
R. White , J. Jose and I. Ruthven, Using top-ranking sentences to facilitate effective information access, Journal of the American Society for Information Science and Technology56 ( 2005) 1113-1125.
22.
T. Paek, S. Dumais, and R. Logan, WaveLens: a new view onto Internet search results. In: E. Dyckstra-Errickson and M. Tscheligi (eds), Proceedings of SIGCHI Conference on Human Factors in Computing Systems (CHI2004) (ACM Press , New York, 2004).
23.
V. Mordechai , O. Drori and A.J. Frank, TTLS: a grouped display of search results based on organisational taxonomy using the LCC&K interface, Proceedings of the Tenth International Conference on Enterprise Information Systems (ICEIS Press, 2008). Available at: www.global-report.com/drori/?l=he&a=291330 (accessed 22 July 2008).
24.
S. Dziadosz and R. Chandrasekar, Do thumbnail previews help people make better decisions about web search results? In: K. Jarvelin, M. Beaulieu, R. Baeza-Yates and S. H. Myaeng (eds), Proceedings of the 25th Annual International Conference on Research and Development in Information Retrieval (ACM SIGIR) (ACM Press, New York, 2002).
25.
J. Vegas , F. Crestani and P. de la Fuente, Context representation for web search results, Journal of Information Science33(1) (2007) 77-94.
26.
O. Drori, Display of search results in Google-based Yahoo! vs LCC&K interfaces: a comparison study, Proceedings of Informing Science 2003 Conference (Informing Science, 2003).
27.
O. Drori and E. Tamir, Display of search results in Hebrew: a comparison study between Google and LCC&K interface, Journal of Information Science31(3) (2005) 164-177.
28.
N. Alon and O. Drori, Using documents classification for displaying search results list, Journal of Information Science29(2) (2003) 106-97.
29.
S. Green , G. Marchionini, C. Plaisant and B. Shneiderman, Previews and overviews in digital libraries: designing surrogates to support visual information seeking, Journal of the American Society for Information Science and Technology51(4) (2000) 380-393.
30.
B. Kules, J. Kustanowitz and B. Shneiderman, Categorizing web search results into meaningful and stable categories using fast-feature techniques, Proceedings of the 6th ACM/IEEE Joint Conference on Digital Libraries-CS, 1-15 June 2006 (ACM Press, New York, 2006).
31.
S. Dumais, E. Cutrell and H. Chen, Optimising search by showing results, Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI 2001 (ACM Press, New York, 2001).
32.
P. Balatsoukas , A. O’Brien and A. Morris, The usability of metadata surrogates in search result interfaces of learning object repositories . In: H. Weghorn and A.P. Abraham (eds), IADIS International Conference: Informatics 2008 (IADIS, Amsterdam, 2008).
33.
E. Wood, Task complexity: definition of the construct. Organisational Behaviour and Human Decision Processes37 ( 1986) 60-82.
34.
I. Ruthven , M. Baillie, L. Azzopardi, R. Bierig, E. Nicol, S. Sweeney and M. Yaciki, Contextual factors affecting the utility of surrogates within exploratory search, Information Processing and Management44(2) (2008) 437-462.
35.
J. Rubin and D. Chisnell, Handbook of Usability Testing (Wiley, Indianapolis, IN, 2007).
36.
M. Eisenberg and L. Barry, Order effects: a study of the possible influence of presentation order on user judgments of document relevance, Journal of the American Society for Information Science39 (1988) 292-300.
37.
Y. Xu and D. Wang, Order effect in relevance judgment, Journal of the American Society for Information Science and Technology59 (2008) 1264-1275.
38.
P. Borlund, The IIR evaluation model: a framework for evaluation of interactive information retrieval systems, Information Research8(3) (2003). Available at: http://informationr.net/ir/8-3/paper152.html
39.
L. Campbell, UK LOM Core update (2004). Available at: http://metadata.cetis.ac.uk/sig_meetings/glasgow2004_presentations/lorna.uklomcore.ppt (accessed 20 October 2009).
40.
N. Friesen, Final report on the ‘international LOM survey’. ISO/IEC JTCI/SC36 (2004). Available at: http://jtc1sc36.org/doc/36N0871.pdf (accessed 19 June 2006).
41.
E. Rosch and B. Lloyd, Cognition and Categorisation (Lawrence Erlbaum, Philadelphia, PA, 1978)
42.
B. Mehlenbacher , M. Duffy and J. Palmer, Finding information on a menu: linking menu organisation to users’ goals, Journal of Human Computer Interaction4 (1989) 231-251.
43.
B. Liu , G. Francis and G. Salvendy, Applying models of visual search to menu design, International Journal of Human-Computer Studies56 (2002) 307-330.
44.
R. Sternberg , Cognitive Psychology (Thompson , London, 2003).
