At some stage, most of the models and techniques implemented in information retrieval use frequency counts of the terms appearing in documents and in queries. However, many words, since they are derived from the same stem, have very close semantic content. This makes a grouping of such variants under a single term advisable. Otherwise, dispersal occurs in the calculation of frequency of these terms and it also becomes difficult to compare queries and documents. On the other hand, there are notable differences between different languages in the way of forming derivatives and inflected forms, so that the application of specific techniques can produce unequal results according to the language of the documents and queries. A description is given of tests carried out for documents in Spanish, which involved some stemming techniques widely used in English, as well as the application of n-grams, and the results are compared.
Get full access to this article
View all access options for this article.
References
1.
[1] G. Salton , Automatic Information Organization and Retrieval ( McGraw-Hill, New York , 1968).
2.
[2] M.F. Porter , An algorithm for suffix stripping , Program14(3) (1980) 130-137 .
3.
[3] C.D. Paice , Method for evaluation of stemming algorithms based on error counting , Journal of the American Society for Information Science47(8) (1996) 632-649 .
4.
[4] J.B. Lovin , Development of a stemming algorithm , Mechanical Translations and Computational Linguistics11(1-2) (1968) 22-31 .
5.
[5] J. Dawson , Suffix removal and word conflation , ALLC Bulletin2(3) (1974) 33-46 .
6.
[6] C. Paice , Another stemmer , ACM SIGIR Forum24(3) (1990) 56-61 .
7.
[7] R. Schinke , A. Robertson , P. Willett and M. Greengrass , A stemming algorithm for Latin text databases , Journal of Documentation52(2) (1996) 172-187 .
8.
[8] F. Ahamad , M. Yussof and M.T. Sembok , Experiments with a stemming algorithm for Malay words , Journal of the American Society for Information Science47(12) (1996) 909-918 .
9.
[9] J. Savoy , Stemming of French words based on grammatical categories , Journal of the American Society for Information Science44(1) (1993) 1-9 .
10.
[10] J. Savoy , A stemming procedure and stopword list for general French corpora , Journal of the American Society for Information Science50(10) (1999) 944-952 .
11.
[11] H. Abu-Salem , M. Al-Omari and M.W. Evens , Stemming methodologies over individual query words for an Arabian information retrieval system , Journal of American Society for Information Science50(6) (1999) 524-529 .
12.
[12] A.M. Robertson and P. Willett , Applications of n-grams in textual information systems , Journal of Documentation54(1) (1998) 48-69 .
13.
[13] J.J. Pollock and A. Zamora , System design for detection and correction of spelling errors in scientific and scholarly text , Journal of American Society for Information Science35 (1984) 104-109 .
14.
[14] G.W. Adamson and J. Boreham , The use of an association measure based on character structure to identify semantically related pairs of words and document titles , Information Storage and Retrieval10 (1974) 253-260 .
15.
[15] M. Lennon , D.S. Peirce , B.D. Tarry and P. Willett , An evaluation of some conflation algorithms for information retrieval , Journal of Information Science3 (1981) 177-183 .
16.
[16] W.B. Cavnar , Using an N-Gram Based Document Representation with a Vector Processing Retrieval Model, TREC-3 (NIST Special Publication 500-226) ( Department of Commerce, National Institute of Standards and Technology, Gaithersburg, MD , 1994). Availableat:http://trec.nist.gov.pubs/trec3/papers/cavnar_ngram_94.ps
17.
[17] M. Damashek , Gauging similarity with n-grams: language independent categorisation of text , Science267 (1995) 843-848 .
18.
[18] S. Huffman , Acquaintance: Language-Independent Document Categorization by N-Grams, TREC-4 (NIST Special Publication 500-236) ( Department of Commerce, National Institute of Standards and Technology, Gaithersburg, MD , 1995). Available at: http://trec.nist.gov/pubs/trec4/papers/nsa.ps
19.
[19] D. Harman , The TREC conferences . In: Proceedings HIM ’95 (Hypertext-Information Retrieval-Multimedia) (UVK, Konstanz, 1995), pp. 9-23 .
20.
[20] D.K. Harman (ed.), Overview of the Third Text Retrieval Conference (TREC-3) (NIST Special Publication 500-226) (Department of Commerce, National Institute of Standards and Technology, Gaithersburg, MD, 1994). Available at: http://trec.nist.gov/pubs/trec3/tc_proceedings.html
21.
[21] D.K. Harman (ed.), The Fourth Text Retrieval Conference (TREC-4) (NIST Special Publication 500-236) (Department of Commerce, National Institute of Standards and Technology, Gaithersburg, MD, 1995). Available at: http://trec.nist.gov/pubs/trec4/t4_proceedings.html
22.
[22] C. Buckley , G. Salton , J. Allan and A. Singhal , Automatic Query Expansion Using SMART: TREC3 (NIST Special Publication 500-226) ( Department of Commerce, National Institute of Standards and Technology, Gaithersburg, MD , 1994). Available at: http://trec.nist.gov/pubs/trec3/papers/cornell.new.ps
23.
[23] R. Gómez Díaz, La Recuperación de Información en Español: Evaluación del Efecto de Sus Peculiaridades Lingüísticas (Unpublished paper, Universidad de Salamanca, Salamanca, 1998).
24.
[24] D. Harman , Ranking algorithms. In: W.B. Frakes and R. Baeza-Yates (eds), Information Retrieval: Data Structures and Algorithms ( Prentice-Hall, Upper Saddle River, NJ , 1992), pp. 363-392.
25.
[25] G. Salton and M.J. McGill , Introduction to Modern Information Retrieval ( McGraw-Hill, New York , 1983).
26.
[26] Real Academia Española, Diccionario de la Lengua Española (RAE, Madrid, 1996).
27.
[27] M. Moliner, Diccionario de Uso del Español (Ed. Gredos, Madrid, 1991).
28.
[28] M.F. Pérez Lagos, Formación de Palabras, la Composición Culta en los Diccionarios (Universidad de Malaga, 1996).
29.
[29] J. V. Muñoz Rodríguez and I. Gil Leiva, Análisis de los descriptores de diferentes áreas del conocimiento indizadas en bases de datos del CSIC. Aplicación de la indización automática, Revista Española de Documentación Científica 20(2) (1997) 150-160.
