A Comparative Study on Feature Selection in Text Categorization

A Comparative Study on Feature Selection in Text Categorization (1997)

Cached

Download Links

[www.cs.cmu.edu]

Other Repositories/Bibliography

CiteULike
DBLP

by Yiming Yang , Jan O. Pedersen

Citations:

739 - 11 self

Versions

Version History

Metadata Version 3

User correction supplied by SystemCorrections

Datum	Value	Source
TITLE	A Comparative Study on Feature Selection in Text Categorization	user correction - Legacy Corrections
AUTHOR NAME	Yiming Yang	user correction - Legacy Corrections
AUTHOR AFFIL	School of Computer Science; Carnegie Mellon University	user correction - Legacy Corrections
AUTHOR ADDR	Pittsburgh, PA 15213-3702, USA	user correction - Legacy Corrections
AUTHOR NAME	Jan O. Pedersen	user correction - Legacy Corrections
AUTHOR AFFIL	; Verity, Inc.; 2	user correction - Legacy Corrections
AUTHOR ADDR	; 894 Ross Dr.; Sunnyvale, CA 94089, USA	user correction - Legacy Corrections
ABSTRACT	This paper is a comparative study of feature selection methods in statistical learning of text categorization. The focus is on aggressive dimensionality reduction. Five methods were evaluated, including term selection based on document frequency (DF), information gain (IG), mutual information (MI), a Ø 2 -test (CHI), and term strength (TS). We found IG and CHI most effective in our experiments. Using IG thresholding with a knearest neighbor classifier on the Reuters corpus, removal of up to 98% removal of unique terms actually yielded an improved classification accuracy (measured by average precision) . DF thresholding performed similarly. Indeed we found strong correlations between the DF, IG and CHI values of a term. This suggests that DF thresholding, the simplest method with the lowest cost in computation, can be reliably used instead of IG or CHI when the computation of these measures are too expensive. TS compares favorably with the other methods with up to 50% vocabulary redu...	user correction - Legacy Corrections
YEAR	1997	INFERENCE
VENUE TYPE	CONFERENCE	INFERENCE
PAGES	412--420	INFERENCE
CITATIONS	28 found	ParsCit 1.0