Soft Margins for AdaBoost

Soft Margins for AdaBoost (1998)

Cached

Download Links

Other Repositories/Bibliography

DBLP

by Gunnar Rätsch , Takashi Onoda , Klaus-R. Müller

Citations:

325 - 22 self

BibTeX

@MISC{Rätsch98softmargins,
    author = {Gunnar Rätsch and Takashi Onoda and Klaus-R. Müller},
    title = {Soft Margins for AdaBoost},
    year = {1998}
}

OpenURL

Abstract

Recently ensemble methods like AdaBoost were successfully applied to character recognition tasks, seemingly defying the problems of overfitting. This paper shows that although AdaBoost rarely overfits in the low noise regime it clearly does so for higher noise levels. Central for understanding this fact is the margin distribution and we find that AdaBoost achieves -- doing gradient descent in an error function with respect to the margin -- asymptotically a hard margin distribution, i.e. the algorithm concentrates its resources on a few hard-to-learn patterns (here an interesting overlap emerge to Support Vectors). This is clearly a sub-optimal strategy in the noisy case, and regularization, i.e. a mistrust in the data, must be introduced in the algorithm to alleviate the distortions that a difficult pattern (e.g. outliers) can cause to the margin distribution. We propose several regularization methods and generalizations of the original AdaBoost algorithm to achieve a soft margin -- a ...

Keyphrases

soft margin margin distribution noise level low noise regime hard margin distribution recognition task interesting overlap emerge several regularization method error function noisy case gradient descent sub-optimal strategy support vector hard-to-learn pattern difficult pattern original adaboost algorithm ensemble method