Abstract

Most information retrieval systems use stopword lists and stemming algorithms. However, we have found that recognizing singular and plural nouns, verb forms, negation, and prepositions can produce dramatically different text classification results. We present results from text classification experiments that compare relevancy signatures, which use local linguistic context, with corresponding indexing terms that do not. In two different domains, relevancy signatures produced better results than the simple indexing terms. These experiments suggest that stopword lists and stemming algorithms may remove or conflate many words that could be used to create more effective indexing terms. Introduction Most information retrieval systems use a stopword list to prevent common words from being used as indexing terms. Highly frequent words, such as determiners and prepositions, are not considered to be content words because they appear in virtually every document. Stopword lists are almost univer...

Citations

183	Automatically Constructing a Dictionary for Information Extraction Tasks - Riloff - 1993
118	How effective is suffixing - Harman - 1991
102	Information extraction as a basis for high-precision text classification - Riloff, Lehnert - 1994
68	The Effectiveness of a Nonsyntactic Approach to Automatic Phrase Indexing for Document Retrieval - Fagan - 1989
45	Symbolic/subsymbolic sentence analysis: Exploiting the best of two worlds - Lehnert - 1991
31	Fully Automatic Syntactically Based Indexing System - Dillon, Gray
2	The DARPA Tipster Project. SIGIR Forum 26(2):26--28 - Harman - 1992
2	Viewing Morphology as an Inference Process. Computer science technical report - Krovetz - 1993