Abstract

A method of drawing index terms from text is presented. The approach uses no stop list, stemmer, or other language-and domain-specific component, allowing operation in any language or domain with only trivial modification. The method uses n-gram counts, achieving a function similar to, but more general than, a stemmer. The generated index terms, which the author calls “highlights, ” are suitable for identifying the topic for perusal and selection. An extension is also described and demonstrated which selects index terms to represent a subset of documents, distinguishing them from the corpus. Some experimental results are presented, showing operation in English, Spanish, German, Georgian, Russian, and Japanese.

Citations

1216	Term-weighting approaches in automatic text retrieval - Salton, Buckley - 1988
332	Discrete Multivariate Analysis: Theory and Practice - Bishop, Fienberg, et al. - 1975
247	A statistical interpretation of term specificity and its application in retrieval - Jones - 1972
156	A technique for computer detection and correction of spelling errors - Damerau - 1964
98	A Statistical Approach to Mechanized Encoding and Searching of Literary Information - Luhn - 1957
77	Principles of multivariate analysis: A user’s perspective - Krzanowski - 1990
73	A probabilistic approach to automatic keyword indexing - Harter - 1975
53	Automatic Spelling Correction Using a Trigram Similarity Measure - Angell, Freund, et al. - 1983
47	The Effectiveness of Stemming For Natural-Language Access to Slovene Textual Data - Popovic, Willett - 1992
45	The reactive keyboard: A predictive typing aid - Darragh, Witten, et al. - 1990
31	Adaptive language modeling using minimum discriminant estimation - Pietra, Pietra, et al. - 1992
30	MExperiments in text recognition with binary n-gram and viterbi algorithms". I EliI:' Transacticns on /'atum Analysis arui - lIulJ, Srihari - 1982
23	The Viterbi algorithm as an aid in text recognition - Neuhoff - 1975
23	A performance evaluation of similarity measures, document term weighting schemes and representations in a Boolean environment - Noreault, McGill, et al. - 1980
18	I). Unsupervised learning and the information retrieval problem - Scholtes
15	The decline of subject searching: long-term trends and patterns of index use in an online catalog - Larson - 1991
12	The application of the concept of homeosemy to natural language information retrieval - Heer - 1982
12	A binary n-gram technique for automatic correction of substitution, deletion, insertion, and reversal errors in words - Ullmann - 1977
11	One-time complete indexing of text: theory and practice - D'Amore, Mah - 1985
11	An evaluation of the applicability of ranking algorithms to improve the effectiveness of full-text retrieval. II. On the effectiveness of ranking algorithms on full-text retrieval - Ro - 1988
10	N-Gram-Based Text Filtering for TREC-2 - Cavnar - 1994
7	Automatic abstracting and indexing-Survey and recommendations - Edmundson, Wyllys - 1961
6	An experiment in automatic indexing - DAMERAU - 1965
5	Hybrid grammar-bigram speech recognition system with first order dependence model - WRIGHT, JONES, et al. - 1992
5	Automatic Detection and Correction of Spelling Errors in a Large Data - Zamora - 1980
4	Context in Word Recognition - Hanson, Riseman, et al. - 1976
4	Variety generation—A reinterpretation of Shannon’s mathematical theory of communication, and its implications for information science - Lynch - 1977
4	Classification of ranking algorithms - Sager, Lockemann - 1976
4	Document retrieval experiments using indexing vocabularies of varying size. II. Hashing, truncation, digram and trigram encoding of index terms - Willet - 1979
3	Document retrieval experiments using indexing vocabularies of varying size. 1. Variety generation symbols assigned to the fronts of index terms - Burnett, Cooper, et al. - 1979
3	A statistical method of spelling correction - Cornew - 1968
3	Generalized linear models - A, Wedderburn - 1972
2	Computer selection of keywords using word-frequency analysis - Carrel, Roeloffs - 1969
2	The degarbler - a program for correcting machine-read morse code - McElwain, Evens - 1962
2	AUlomatic text processing - unknown authors - 1988
2	A simplified heuristic version of a recursive Bayes algorithm for using context in text recognition - Shinghal, Rosenberg, et al. - 1978
1	Mathematical statisfics - Bickel, Doksum - 1977
1	Introduction Co algorithms - Cormen, Leiserson, et al. - 1990
1	Suboptimal sequential decision schemes with on-line feature ordering - Hussain, Donaldson - 1974
1	The art qfcomputerprogramming (vol - Knuth - 1973
1	Accelerated text searching through signature trees - L - 1990
1	A practical approach to data structures - Lester - 1990
1	28). DISCIPLEfinal report - Mah, D’Amore - 1983
1	Computer detection oftypographical errors - Morris, Cherry - 1975
1	A study ofEnglish word category prediction based on neural networks - M, Shikano - 1995
1	Continuous-speech recognition using a stochastic language model - unknown authors - 1989
1	A contextual postprocessing system for error correction using binary n-grams - R - 1974
1	A theory ofterm importance in automatic text analysis - S, Yu - 1975
1	Keypapers in information science. Washington, DC: American Society for Information Science - unknown authors - 1980
1	Selection ofequifrequent word fragments for information retrieval - Schuegraf, Heaps - 1973