A Comparative Study on Feature Selection in Text Categorization

DMCA

A Comparative Study on Feature Selection in Text Categorization (1997)

Cached

Download Links

[www.cs.cmu.edu]
[nyc.lti.cs.cmu.edu]

Other Repositories/Bibliography

CiteULike
DBLP

by Yiming Yang , Jan O. Pedersen

Citations:

1319 - 15 self

BibTeX

@INPROCEEDINGS{Yang97acomparative,
    author = {Yiming Yang and Jan O. Pedersen},
    title = {A Comparative Study on Feature Selection in Text Categorization},
    booktitle = {},
    year = {1997},
    pages = {412--420},
    publisher = {Morgan Kaufmann Publishers}
}

OpenURL

Abstract

This paper is a comparative study of feature selection methods in statistical learning of text categorization. The focus is on aggressive dimensionality reduction. Five methods were evaluated, including term selection based on document frequency (DF), information gain (IG), mutual information (MI), a Ø 2 -test (CHI), and term strength (TS). We found IG and CHI most effective in our experiments. Using IG thresholding with a knearest neighbor classifier on the Reuters corpus, removal of up to 98% removal of unique terms actually yielded an improved classification accuracy (measured by average precision) . DF thresholding performed similarly. Indeed we found strong correlations between the DF, IG and CHI values of a term. This suggests that DF thresholding, the simplest method with the lowest cost in computation, can be reliably used instead of IG or CHI when the computation of these measures are too expensive. TS compares favorably with the other methods with up to 50% vocabulary redu...

Keyphrases

text categorization comparative study feature selection knearest neighbor classifier mutual information reuters corpus df thresholding feature selection method improved classification accuracy term selection vocabulary redu document frequency unique term chi value term strength aggressive dimensionality reduction average precision statistical learning strong correlation information gain

DMCA