Abstract

This paper proposes and analyzes an approach to estimating the generalization performance of a support vector machine (SVM) for text classification. Without any computation intensive resampling, the new estimators are computationally much more ecient than cross-validation or bootstrap, since they can be computed immediately from the form of the hypothesis returned by the SVM. Moreover, the estimators delevoped here address the special performance measures needed for text classification. While they can be used to estimate error rate, one can also estimate the recall, the precision, and the F 1 . A theoretical analysis and experiments on three text classification collections show that the new method can effectively estimate the performance of SVM text classifiers in a very efficient way.

Citations

6696	The Nature of Statistical Learning Theory - Vapnik - 1995
1783	An Introduction to the Bootstrap - Tibshirani, R - 1993
1491	Support-vector networks - Cortes, Vapnik - 1995
1368	Text categorization with support vector machines - Joachims - 1998
1216	Term-weighting approaches in automatic text retrieval - Salton, Buckley - 1988
1086	Making large-scale SVM learning practical - Joachims - 1999
882	Practical Methods of Optimization - Fletcher - 1987
801	A probabilistic theory of pattern recognition - Devroye, Györfi, et al. - 1996
688	Estimation of dependencies based on empirical data - Vapnik - 1982
528	A study of cross-validation and bootstrap for accuracy estimation and model selection - Kohavi - 1995
444	The jackknife, the bootstrap and other resampling plans - Efron - 1982
419	Inductive learning algorithms and representations for text categorization - Dumais, Platt, et al. - 1998
174	Dongsheeng T. The Jackknife and Bootstrap - Shao - 1995
148	Estimating the error rate of a prediction rule: improvement on cross-validation - Efron - 1983
140	Learning to classify text from labeled and unlabeled documents - Nigam, McCallum, et al. - 1998
122	Support Vector Machines, Reproducing Kernel Hilbert Spaces and the randomized GACV - Wahba - 1997
110	Probability inequalities for sums of bounded random variables - Hoeding - 1963
101	An Experimental and Theoretical Comparison of Model Selection Methods - Kearns, Mansour, et al. - 1999
90	Probabilistic kernel regression models - Jaakkola, Haussler - 1999
76	Algorithmic stability and sanity-check bounds for leave-one-out cross-validation - Kearns, Ron - 1999
69	Estimation of error rates in discriminant analysis, Technometrics 10 - Lachenbruch, Mickey - 1968
45	Distribution-Free performance Bounds for Potential Function Rules - Devroye, L, et al. - 1979
38	Asymptotics for and against cross-validation - Stone - 1977
29	Cross-validatory choice and assesment of statistical predictions - Stone - 1974
27	Uniqueness of the svm solution - Burges, Crisp - 1999
23	A bound on the error of cross validation using the approximation and estimation rates, with consequences for the trainingtest split - Kearns - 1997
18	Estimating the accuracy of learned concepts - Bailey, C - 1993
18	A finite sample distribution-free performance bound for local discrimination rules - Rogers, Wagner - 1978
17	Estimating the generalization performance of a SVM eciently - Joachims - 2000
13	Distribution-free inequalities for the deleted and holdout error estimates - Devroye, Wagner - 1979
11	Theorie der Zeichenerkennung - Wapnik, Tscherwonenkis - 1979
10	A probabilistic theory of pattern recognition - L, Lugosi - 1979
10	Classi� cation and Regression T - Breiman, Friedman, et al. - 1984
8	Evaluation of attributes obtained in statistical decision rules - Lunts, Brailovsky - 1967
6	A distribution-free performance bound in error estimation - Devroye, Wagner - 1976
4	A traininig algorithm for optimal margin classifiers - Boser, Guyon, et al. - 1992
4	A Note on Support Vector Machine Degeneracy - Rifkin, Pontil, et al. - 1999
3	On the bias and variability of bootstrap and cross-validation estimates of error rate in discriminant problems. Biometrika - Davison, Hall - 1992
3	Algorithms for recognizing contour-traced handprinted characters - Toussaint, Donaldson - 1970
2	Asymptotics for and against cross-validation. Biometrika - Stone - 1977
1	Automatic model selection for svms in text classification - Joachims - 2000