IR researchers need to collect comparative data to evaluate the worth of possible system improvements, for example new techniques for query enhancement. However it is sometimes difficult to achieve strict comparability in experiments designed for an interactive environment. We describe a method for merging results from different queries and collecting relevance feedback from users in comparative experiments. The method has been implemented in scripts supporting a Web-based front-end to a probabilistic retrieval system, using a relational database to log results. Its practical use is demonstrated in an investigation of initial query enhancement with collocate terms.
Get full access to this article
View all access options for this article.
References
1.
[1] S. Robertson , Overview of the Okapi projects , Journal of Documentation53(1) (1997), 3-7 .
2.
[2] K. Sparck Jones , S. Walker and S.E. Robertson , A probabilistic model of information retrieval: development and comparative experiments , Information Processing and Management36 (2000) 779-808 and 809-840 .
3.
[3] A. Goker , Context learning in Okapi , Journal of Documentation53(1) (1997) 80-83 .
4.
[4] S. Jones , M. Gatfordet al. , Thesaurus-based query enhancement. In: E.Atwell (ed.), Knowledge at Work in Universities ( Leeds University Press , 1993), pp. 15-19.
5.
[5] O. Vechtomova and S. Robertson , Integration of collocation statistics into the probabilistic retrieval model , Proceedings of the 22nd BCS IRSG Meeting (Cambridge, April 2000), pp. 165-177 .
6.
[6] M. Karamuftuogluet al. , Challenges posed by Web-based document retrieval: participation of Okapi in TIPS , Journal of Information Science28(1), 2002, 3-17 .
7.
[7] S. Robertson , S. Walkeret al. , Okapi at TREC 3 . In: Overview of the Third Text Retrieval Conference (TREC-3), NIST Special Publication 500-226 (1995), pp. 109-126 . Available at: http://trec.nist.gov/pubs/trec3/t3_proceedings.html
8.
[8] E. Vorhees , Siemens TREC-4 report: further experiments with database merging . In: The Fourth Text Retrieval Conference (TREC-4), NIST Special Publication 500-236 (1996), pp. 121-130 . Available at: http://trec.nist.gov/pubs/trec4/t4_proceedings.html
9.
[9] T. Saracevic , Relevance: a review of and a framework for the thinking on the notion in information science . Journal of the American Society for Information Science26(6) (1975), 321-343 .
10.
[10] O. Vechtomova, Approaches to using word collocation in information retrieval. Ph.D. Thesis (City University, 2001).
11.
[11] T. McEnery and A. Wilson , Corpus Linguistics ( Edinburgh , 1996).
