A PAC-Bayesian Margin Bound for Linear Classifiers

DMCA

A PAC-Bayesian Margin Bound for Linear Classifiers (2002)

Cached

Download Links

Other Repositories/Bibliography

DBLP

by Ralf Herbrich , Thore Graepel

Citations:

37 - 3 self

BibTeX

@MISC{Herbrich02apac-bayesian,
    author = {Ralf Herbrich and Thore Graepel},
    title = {A PAC-Bayesian Margin Bound for Linear Classifiers},
    year = {2002}
}

OpenURL

Abstract

We present a bound on the generalisation error of linear classifiers in terms of a refined margin quantity on the training sample. The result is obtained in a PAC-Bayesian framework and is based on geometrical arguments in the space of linear classifiers. The new bound constitutes an exponential improvement of the so far tightest margin bound, which was developed in the luckiness framework, and scales logarithmically in the inverse margin. Even in the case of less training examples than input dimensions sufficiently large margins lead to non-trivial bound values and---for maximum margins---to a vanishing complexity term. In contrast to previous results, however, the new bound does depend on the dimensionality of feature space. The analysis shows that the classical margin is too coarse a measure for the essential quantity that controls the generalisation error: the fraction of hypothesis space consistent with the training sample. The practical relevance of the result lies in the fact that the well-known support vector machine is optimal with respect to the new bound only if the feature vectors in the training sample are all of the same length. As a consequence we recommend to use SVMs on normalised feature vectors only. Numerical simulations support this recommendation and demonstrate that the new error bound can be used for the purpose of model selection.

Keyphrases

linear classifier pac-bayesian margin bound training sample new bound generalisation error feature space numerical simulation support complexity term pac-bayesian framework maximum margin previous result essential quantity new error bound luckiness framework normalised feature vector hypothesis space consistent well-known support vector machine practical relevance tightest margin bound geometrical argument input dimension exponential improvement non-trivial bound value classical margin training example model selection feature vector large margin inverse margin refined margin quantity

DMCA