Abstract

In this article we study the generalization abilities of several classifiers of support vector machine (SVM) type using a certain class of kernels that we call universal. It is shown that the soft margin algorithms with universal kernels are consistent for a large class of classification problems including some kind of noisy tasks provided that the regularization parameter is chosen well. In particular we derive a simple su#cient condition for this parameter in the case of Gaussian RBF kernels. On the one hand our considerations are based on an investigation of an approximation property---the so-called universality---of the used kernels that ensures that all continuous functions can be approximated by certain kernel expressions. This approximation property also gives a new insight into the role of kernels in these and other algorithms. On the other hand the results are achieved by a precise study of the underlying optimization problems of the classifiers. Furthermore, we show consistency for the maximal margin classifier as well as for the soft margin SVM's in the presence of large margins. In this case it turns out that also constant regularization parameters ensure consistency for the soft margin SVM's. Finally we prove that even for simple, noise free classification problems SVM's with polynomial kernels can behave arbitrarily badly.

Citations

1057	An Introduction to Support Vector Machines and other kernel-based learning methods - Cristianini, Shawe-Taylor - 2000
801	A probabilistic theory of pattern recognition - Devroye, Györfi, et al. - 1996
295	Real Analysis and Probability - Dudley - 2002
239	Weak Convergence and Empirical Processes - VAART, A - 1996
221	Structural risk minimization over data-dependent hierarchies - Shawe-Taylor, Bartlett, et al. - 1940
110	Central limit theorems for empirical measures - Dudley - 1979
105	Generalization performance of support vector machines and other pattern classifiers - Bartlett, Shawe-Taylor, et al. - 1999
96	A Generalized Representer Theorem - Schölkopf, Herbrich, et al.
51	Entropy, compactness and the approximation of operators - Carl, Stephani - 1990
41	On optimal nonlinear associative recall - Poggio - 1975
38	Analysis Now - Pedersen - 1989
27	Uniqueness of the svm solution - Burges, Crisp - 1999
12	A generalized representer theorem - Schölkpf, Herbrich, et al.
7	On the generalization ability of support vector machines - Steinwart - 2001
6	Theory of support vector machines - Stitson, Wetson, et al. - 1996
6	A leave-one-out cross validation bound for kernel methods with applications in learning - Zhang - 2001
5	Ryshik: Summen-, Produkt und Integral-Tafeln - Gradstein, M - 1981
4	On the influence of the kernel on the generalization ability of support vector machines - Steinwart - 2001