The paper presents a new method for handling of morphological variation of query terms in best-match IR. The method is based on enhanced inflectional stems. Use of inflectional stems has earlier been shown to be a good retrieval method in inflected indexes in a best-match environment for a highly inflected and compound-rich language, Finnish. In this paper the earlier stem method is elaborated upon by enhancing the stems with regular expressions. Contrary to our expectations the results show that the enhanced stem queries do not outperform basic inflectional stems, but neither are they considerably worse with long queries. With short web-like queries they perform relatively better than with long queries and outperform clearly stemming (Finnish stemmer of Snowball) and plain, unprocessed query words. The main benefits of the proposed method, besides fairly good precision and recall (P-R) performance, are shorter and more manageable queries, which is of practical importance, e.g. with large web indexes.
J.B. Lovins , Development of a stemming algorithm , Mechanical Translation and Computational Linguistics11 (1968) 22–31 .
2.
W. B. Frakes , Stemming algorithms. In: W.B. Frakes and R. Baeza-Yates (eds), Information Retrieval. Data Structures and Algorithms ( Prentice Hall, Englewood Cliffs, MA , 1992).
3.
R. Sproat , Computation and Morphology ( MIT Press, Cambridge, MA , 1992).
4.
R. Alkula , Merkkijonoista Suomen Kielen Sanoiksi(2000). Acta Universitatis Tamperensis763-763. Available at: http://acta.uta.fi/pdf/951-44-4886-3.pdf (accessed 6 January 2006).
5.
R. Alkula , From plain character strings to meaningful words: producing better full text databases for inflectional and compounding languages with morphological analysis software , Information Retrieval4 (2001) 195–208 .
6.
V. Hollink , J. Kamps , C. Monz and M. de Rijke , Mono-lingual document retrieval for European languages , Information Retrieval7 (2004) 33–52 .
7.
W. Kraaij , Variations on Language Modeling for Information Retrieval (CTIT Ph. D. series No. 04–62, Den Haag, 2004).
8.
N. Alemayehu and P. Willett , Stemming of Amharic words for information retrieval , Literary and Linguistic Computing17 (2002) 1–17 .
9.
N. Alemayehu and P. Willett , The effectiveness of stemming for information retrieval in Amharic , Program37 (2003) 254–259 .
10.
H. Abu-Salem , M. Al-Omari and M.W. Evens , Stemming methodologies over individual query words for an Arabic information retrieval system , Journal of the American Society for Information Science50 (1999) 524–529 .
11.
M. Braschler and B. Ripplinger , How effective is stemming and decompounding for German text retrieval?Information Retrieval7 (2004) 291–316 .
12.
M. Popovic<caron > and P. Willett , Processing of documents and queries in a Slovene language free text retrieval system , Literary and Linguistic Computing5 (1990) 182–190 .
13.
M. Popovic<caron > and P. Willett , The effectiveness of stemming for natural-language access to Slovene textual data , Journal of the American Society for Information Science43 (1992) 384–390 .
14.
H. Sever and Y. Bitirim , FindStem: analysis and evaluation of a Turkish stemming algorithm . In: M.A. Nascimento , E.S. de Moura and A.L. Oliveira (eds), String Processing and Information Retrieval. 10th International Symposium, SPIRE 2003, Manaus, Brazil, 2003 (Springer, Heidelberg, 2003) 238–251 .
15.
E. Airio , Word normalization and decompounding in mono-and bilingual IR , Information Retrieval9 (2006) 249–271 .
16.
J. Mayfield and P. McNamee , Single N-gram stemming . In: Proceedings of Sigir2003, The Twenty-Sixth Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, 28 July to 1 August, Toronto (ACM, 2003).
17.
R. KrovetzViewing morphology as an inference process , Artificial Intelligence118 (2000) 277–294 .
18.
G. Silva and C. Oliveira , The implementation and evaluation of a lexicon-based stemmer . In: M.A. Nascimento , E.S. de Moura and A.L. Oliveira (eds), String Processing and Information Retrieval. 10th International Symposium, SPIRE 2003, Manaus, Brazil, 2003 (Springer, Heidelberg, 2003) 266–276 .
19.
S. Tomlinson , Experiments in 8 European Languages with Hummingbird SearchServer(tm) at CLEF 2002. Available at: www.clef-campaign.org/workshop2002/WN/26.pdf (accessed 6 January 2006).
20.
S. Tomlinson , Lexical and Algorithmic Stemming Compared for 9 European Languages with Hummingbird SearchServer(tm) at CLEF 2003. Available at: http://clef.iei.pi.cnr.it/2003/WN_web/19.pdf (accessed 6 January 2006).
21.
K. Kettunen , T. Kunttu and K. Järvelin , To stem or lemmatize a highly inflectional language in a probabilistic IR environment?Journal of Documentation61(4) (2005) 476–496 .
22.
R. Nurminen , Suomen Kielen Sanamuotoja Tulkitsevien Ohjelmien Hyödyntäminen Tiedonhakujärjestelmissä( VTT, Tutkimuksia386-386, Espoo, 1986).
D. Jurafsky and J.H. Martin , Speech and Language Processing ( Pearson Education , 2000). [Low Price Edition]
25.
E. Roche and Y. Schabes (eds), Finite-State Language Processing ( MIT Press, Cambridge, MA , 1997).
26.
D. Hull , Stemming algorithms: a case study for detailed evaluation , Journal of the American Society for Information Science and Technology47 (1996), 70–84 .
27.
F. Karlsson , Suomen KielenÄänne-ja Muotorakenne( WSOY, Helsinki , 1983).
28.
F. Karlsson , Finnish Grammar ( WSOY, Helsinki1983).
29.
A. Niemikorpi , Suomen Kielen Sanaston Dynamiikkaa[Dynamics of the Finnish Vocabulary], Acta Wasaensia 26 (1991).
30.
L. Karttunen , J.-P. Chanod , G. Grefenstette and A. Schiller , Regular expressions for natural language engineering , Natural Language Engineering2 (1997) 305–328 .
31.
J. Kekäläinen , The Effects of Query Complexity, Expansion and Structure on Retrieval Performance in Probabilistic Text Retrieval, Acta Universitatis Tamperensis678-678 (1999).
32.
E. Sormunen , A Method for Measuring Wide Range Performance of Boolean Queries in Full-Text Databases, Acta Universitatis Tamperensis748-748 (2000).
33.
E. Sormunen , The Effectiveness of Free-Text Searching in Full-Text Databases Containing Newspaper Articles and Abstracts ( Technical Research Centre of Finland. Research Publications790-790, Espoo, 1994). [In Finnish, English abstract].
34.
F. Crestani , M. Sanderson , M. Theophylactou and M. Lalmas , Short queries, natural language and spoken document retrieval: experiments at Glasgow University. In: E.M. Voorhees and D.K. Harman (eds), The Sixth Text REtrieval Conference (TREC-6), 667–686. [NIST Special Publication 500–240] Available at: http://trec.nist.gov/pubs/trec6/papers/glasgow.ps.gz (accessed 5 December 2005).
35.
C. Crouch , D.B. Crouch , Q. Chen and S.J. Holtz , Improving the retrieval effectiveness of very short queries , Information Processing and Management38 (2002), 1–36 .
36.
B. Jansen , A. Spink and T. Sarasevic , Real life , real users , and real needs : a study and analysis of user queries on the web , Information Processing and Management36 (2000) 207–227 .
37.
J. Kekäläinen , and K. Järvelin , Using graded relevance assessments in IR evaluation . Journal of the American Society for Information Science and Technology53 (2002) 1120–1129 .
38.
D. Dougherty , Sed & awk ( O'Reilly, Sebastopol, CA , 1992).
39.
B.H. Partee , Fundamentals of Mathematics for Linguistics (D. Reidel, Dordrecht, 1978).
40.
B.H. Partee , A. ter Meulen and R.E. Wall , Mathematical Methods in Linguistics ( Kluwer Academic Press, Dordrecht , 1993). [Corrected second printing of the first edition]
41.
G. Navarro and M. Raffinot , Flexible Pattern Matching in Strings ( Cambridge University Press, Cambridge , 2002).
C.L.A. Clarke , G.V. Cormack and E.A. Tudhope , Relevance ranking for one to three term queries , Information Processing and Management36 (2000) 291–311 .
44.
D. Harman , How effective is suffixing , Journal of the American Society for Information Science and Technology42 (1991) 7–15 .
45.
B. Comrie , Language Universals and Linguistic Typology: Syntax and Morphology ( Blackwell, Oxford , 1993).
46.
A. Pirkola , Morphological typology of languages for IR , Journal of Documentation57 (2001) 330–348 .
