Abstract

We study an approach to text categorization that combines distributional clustering of words and a Support Vector Machine (SVM) classifier. The word-cluster representation is computed using the recently introduced Information Bottleneck method, which generates a compact and e#cient representation of documents. When combined with the classification power of the SVM, this method yields high performance in text categorization. We compare this technique with SVM-based categorization using the simple minded bag-of-words (BOW) representation. The comparison is performed over three known datasets. On one of these datasets (the 20 Newsgroups) the method that is based on word clusters significantly outperforms the word-based representation in terms of categorization accuracy or representation e#ciency. On the two other sets (Reuters-21578 and WebKB) the word-based representation slightly outperforms the word-cluster representation. We investigate the potential reasons for this behavior.

Citations

6696	The Nature of Statistical Learning Theory - Vapnik - 1995
6515	Elements of Information Theory - Cover, Thomas - 1991
6232	Maximum likelihood from incomplete data via the EM algorithm - Dempster, Laird, et al. - 1977
3341	Pattern Classification and Scene Analysis - Duda, Hart - 1973
2699	MJ: Introduction to Modern Information Retrieval - Salton, McGill - 1986
2168	Indexing by latent semantic analysis - Deerwester, Dumais, et al. - 1990
1656	CJC: A tutorial on support vector machines for pattern recognition - Burges - 1998
1491	Support-vector networks - Cortes, Vapnik - 1995
1368	Text categorization with support vector machines - Joachims - 1998
946	Combining labeled and unlabeled data with co-training - Blum, Mitchell - 1998
839	Machine learning in automated text categorization - Sebastiani - 2002
741	Shawe-Taylor J (2000) An Introduction to Support Vector Machines and other kernel-based learning methods - Cristianini
533	X: A re-examination of text categorization methods - Yang, Liu
478	L: Distributional Clustering of English Words - Pereira, Tishby, et al. - 1993
448	Solving Multiclass Learning Problems via Error-Correcting Output Codes - Dietterich, Bakiri - 1995
419	Inductive learning algorithms and representations for text categorization - Dumais, Platt, et al. - 1998
373	BoosTexter: A boosting-based system for text categorization.” Machine learning 39.2 - Schapire, Singer - 2000
345	Making large-scale support vector machine learning practical - Joachims - 1998
332	Reducing multiclass to binary: A unifying approach for margin classifiers - Allwein, Schapire, et al. - 2000
330	The Information Bottleneck Method - Tishby, Pereira, et al. - 1999
301	Toward optimal feature selection - Koller, Sahami - 1996
290	DiPasquo,“Learning to extract Symbolic Knowledge from the World Wide Web - Craven, Freitag, et al. - 1998
282	Text classification using string kernels - Lodhi, Saunders, et al. - 2002
207	Using maximum entropy for text classification - Nigam - 1999
193	Deterministic annealing for clustering, compression, classification, regression, and related optimization problems - Rose - 1998
173	Measures of Distributional Similarity - Lee - 1999
140	Learning to classify text from labeled and unlabeled documents - Nigam, McCallum, et al. - 1998
118	Agglomerative information bottleneck - Slonim, Friedman, et al. - 1999
88	Stork Pattern Classification (2nd ed - Duda, Hart, et al. - 1998
77	Multivariate information bottleneck - Friedman, Mosenzon, et al. - 2001
77	Unsupervised document classification using sequential information maximization - Thesis, Friedman, et al. - 2002
63	Maximizing text-mining performance - Weiss, Apte, et al. - 1999
58	Improving multiclass text classification with the support vector machine - Rennie, Rifkin - 2001
53	The power of word clusters for text classification - Slonim, Tishby - 2001
50	Exploiting structural information for text classification on the WWW - Fürnkranz - 1999
39	On Feature Distributional Clustering for Text Categorization - Bekkerman, El-Yaniv, et al. - 2001
27	O.Souroujon. Iterative double clustering for unsupervised and semi-supervised learning - El-Yaniv - 2001
27	Hypertext categorization using hyperlink patterns and meta data - Ghani, Slattery, et al. - 2001
26	Modern Information Retrieval. AddisonWesley, 2nd edition - Baeza-Yates, Ribeiro-Neto
25	Feature generation using general constructor functions - Markovitch, Rosenstein - 2002
21	A Linear Least Squares Fit Mapping Method for Information Retrieval from Natural Language Texts - Yang, Chute - 1992
17	Estimating the generalization performance of a SVM eciently - Joachims - 2000
14	Joining statistics with NLP for text categorization - Jacobs - 1992
11	Objective classification of galaxy spectra using the information bottleneck method.” Monthly Notes of the Royal Astronomical Society, 323:2001, 2001. [ST01] [SW73] [SW06] [TPB99] Noam Slonim and Naftali Tishby. “The power of word clusters for text classif - Slonim, Somerville, et al. - 2001
10	Language-sensitive text classification - Basili, Moschitti, et al. - 2000
8	Relevance feedback in information retrieval, chapter 14. Relevance Feedback - Rocchio - 1971
6	Axiomatic approach feature subset selection based on relevance - Wang, Bell, et al.
5	Machine learning for information retrieval: Advanced techniques, 2000. A tutorial presented at SIGIR'00 - Singer, Lewis
4	Second order features for maximising text classification performance - Raskutti, Ferra, et al. - 2001
1	Feature Extraction, Construction and Selection : A Data Mining Perspective, chapter The Wrapper Approach - Kohavi, John - 1998