Abstract

We present in this paper Linguini, a vector-space based categorize tailored for high-precision language identlj?cation. We show how the accuracy depends on the size of the input document, the set of languages under consideration, and the features used. We found that Linguini could identifi the language of documents as short as 5-10 % of the size of average Web documents with 100% accuracy. We also present how to determine z ~ a document is in two or more languages, and in what proportions, without incurring any appreciable computational overhead beyond the monolingual analysis. This approach can be applied to subject-categorization systems to distinguish between cases where, when the system recommends two or more categories, the document belongs strongly to all or really to none.

Citations

2699	MJ: Introduction to Modern Information Retrieval - Salton, McGill - 1986
739	A Comparative Study on Feature Selection in Text Categorization - Yang
413	An Evaluation of Statistical Approaches to Text Categorization - Yang - 1999
216	Training algorithms for linear text classi ers - Lewis, Shapire, et al. - 1996
207	Automated learning of decision rules for text categorization - Apte, Damerau, et al. - 1994
133	Expert network: Effective and efficient learning from human decisions in text categorization and retrieval - Yang - 1994
96	Gauging similarity with n-grams: Language-independent categorization of text - Damashek - 1995
56	Pattern Classi®cation and Scene Analysis - Duda, Hart - 1973
16	Searching the World - Lawrence, Giles - 1998
15	Computing optimal attribute weight settings for nearest neighbour algorithms - Ling, Wang - 1997
1	N-Gram-Based Text Categorization”, Symposium On Document Analysis and Information Retrieval - Cavnar, Trenkle - 1994
1	Comparing Two Language Identification Schemes”, http://www.rxrc. xerox.cotipublis/ml ttijadtijadt.html - Grefenstette
1	Ranking Algorithms”, in Information Retrieval: Data Structures and Algorithms edited by W - Harmon - 1992