Although known item searching for music can be dealt with by searching metadata using existing text search techniques, human subjectivity and variability within the music itself make it very difficult to search for unknown items. This paper examines these problems within the context of text retrieval and music information retrieval. The focus is on ascertaining a relationship between music relevance criteria and those relating to relevance judgements in text retrieval. A data-rich collection of relevance judgements by creative professionals searching for unknown musical items to accompany moving images using real world queries is analysed. The participants in our observations are found to take a socio-cognitive approach and use a range of content- and context-based criteria. These criteria correlate strongly with those arising from previous text retrieval studies despite the many differences between music and text in their actual content.
J.S. Downie, Toward the scientific evaluation of music information retrieval systems, Proceedings of International Symposium on Music Information Retrieval , Baltimore, USA, 27-30 October 2003 .
2.
D. Bawden, User-Oriented Evaluation of Information Systems and Services (Gower, Aldershot, 1990).
3.
T. Saracevic , Relevance: a review of the literature and a framework for thinking on the notion in information science. Part II: nature and manifestations of relevance, Journal of the American Society for Information Science and Technology58(13) (2007) 1915-1933.
4.
N. Belkin , R. Oddy and H. Brooks, ASK for information retrieval, Journal of Documentation38 (1982 ) 61-71 (part 1) and 145-164 (part 2).
5.
T. Saracevic , Relevance: a review of the literature and a framework for thinking on the notion in information science. Part III: behaviour and effects of relevance, Journal of the American Society for Information Science and Technology58(13) (2007) 2126-2144.
6.
S. Mizzaro , Relevance: the whole history, Journal of the American Society for Information Science48(9) (1997) 810-832.
7.
L. Schamber , Relevance and information behaviour, Annual Review of Information Science and Technology29 (1994) 3-48
8.
T.P. Park , Toward a theory of user-based relevance: a call for a new paradigm of inquiry, Journal of the American Society for Information Science45(3) (1994) 135-141.
9.
C. Barry, User-defined relevance criteria: an exploratory study, Journal of the American Society for Information Science45(3) (1994) 149-159.
10.
C. Barry and L. Schamber, Users’ criteria for relevance evaluation: a cross-situational comparison, Information Processing and Management34(2/3) (1998) 219-236.
11.
T.P. Park , The nature of relevance in information retrieval: an empirical study, Library Quarterly63(3) (1993) 318-351.
12.
S. Harter , Variations in relevance assessments and the measurement of retrieval effectiveness, Journal of the American Society for Information Science47(1) (1996) 37-49.
13.
E. Cosijn and P. Ingwersen, Dimensions of relevance, Information Processing and Management36 ( 2000) 533-550.
14.
T.D. Anderson , Studying human judgments of relevance: interactions in context. In: Information Interaction in Context Proceedings of 1st IIiX Symposium on Information Interaction in Context (Royal School of Library and Information Science, Copenhagen, Denmark, 18-20 October 2006).
15.
I. Ruthven , Interactive information retrieval, Annual Review of Information Science and Technology42(1) (2008) 43-91.
16.
E. Rasmussen , Evaluation in information retrieval, The MIR/MDL Evaluation Project White Paper Collection Edition Number3 (2003) 45-49. Available at: www.music-ir.org/evaluation/wp.html (accessed 23 March 2010).
17.
E. Sormunen, M. Markkula and K. Järvelin, The perceived similarity of photos - a test-collection based evaluation framework for the content-based image retrieval algorithms, Final Mira Conference, 1999 . In: S. Draper, M.D. DunlopI. Ruthven and C.J. van Rijsbergen (eds), Mira 99: Evaluating interactive information retrieval, Electronic Workshops in Computing (eWic), 1999.
18.
Mirex (2009) MIREX 2005-2008 Wikis, Available at: www.music-ir.org/mirex/2009/index.php/Main_Page#MIREX_2005_-_2008_Wikis (accessed 19 January 2010).
19.
Trec (2010) Text REtrieval Conference. Available at: http://trec.nist.gov/ (accessed 19 January 2010).
20.
E. Voorhees , Whither music IR evaluation infrastructure: lessons to be learned from TREC, The MIR/MDL Evaluation Project White Paper Collection Edition Number3 (2003) 7-13. Available at: www.music-ir.org/evaluation/wp.html (accessed 23 March 2010).
21.
S. Rüger , (2003) A framework for the evaluation of content-based music information retrieval using the TREC paradigm, The MIR/MDL Evaluation Project White Paper Collection Edition Number3 (2003) 68-70. Available at: www.music-ir.org/evaluation/wp.html (accessed 23 March 2010).
22.
J.S. Downie, Interim report on establishing MIR/MDL evaluation frameworks: commentary on consensus building, The MIR/MDL Evaluation Project White Paper Collection Edition Number3 ( 2003) 43-44. Available at: www.music-ir.org/evaluation/wp.html (accessed 23 March 2010).
23.
J.S. Downie, The music information retrieval exchange (2005-2007): a window into music information retrieval research, Acoustic Science and Technology29(4) ( 2008) 247-255.
24.
A. Flexer, Statistical evaluation of music information retrieval experiments, Journal of New Music Research35(2) (2006) 113-120.
25.
A. Gruzd, J.S. Downie, M. Cameron Jones, J.H. Lee, Evalutron 6000: collecting music relevance judgments, Proceedings of Joint Conference on Digital Libraries 2007 ( Vancouver, British Columbia, Canada June 17-22 2007).
26.
M. Mandel and D. Ellis, A web-based game for collecting music metadata, Proceedings of 8th International Conference on Music Information Retrieval (Vienna, Austria, 23-27 September 2007 ).
27.
L. Barrington , R. Oda and G. Lanckreit, Smarter than genius?Human evaluation of music recommender systems, Proceedings of 10th International Society for Music Information Retrieval ( Kobe, Japan, October 26-30 2009).
28.
E. Law and L. von Ahn, Input-agreement: a new mechanism for collecting data using human computation games, Proceedings of CHI 2009 ( Boston, Massachusetts, USA, April 4-9 2009).
29.
C. Inskip, A. Macfarlane and P. Rafferty, Organizing music for movies, Proceedings of International Society for Knowledge Organization (UK) Content Architecture Conference ( London, 22-23 June 2009).
30.
Spotify (2009) A World of Music. Available at: www.spotify.com/en/ (accessed 18 January 2010).
31.
C. Inskip, A. Macfarlane and P. Rafferty, Towards the disintermediation of creative music search: analysing queries to determine important facets, Proceedings of ECDL Workshop on Exploring Musical Information Spaces (Corfu , Greece, 1-2 October 2009).
32.
J. Kim and N. Belkin, Categories of music description and search terms and phrases used by non-music experts, Proceedings of 3rd International Conference on Music Information Retrieval IRCAM (Paris, France, 13-17 October 2002).
33.
P. Borlund , The concept of relevance in IR, Journal of the American Society for Information Science and Technology54(10) (2003) 913-925.
34.
C. Inskip , A. Macfarlane and P. Rafferty, Meaning, communication, music: towards a revised communication model, Journal of Documentation64(5) (2008) 687-706.
