Abstract

The theory behind the success of adaptive reweighting and combining algorithms (arcing) such as Adaboost (Freund and Schapire [1995].[1996]) and others in reducing generalization error has not been well understood. By formulating prediction, both classification and regression, as a game where one player makes a selection from instances in the training set and the other a convex linear combination of predictors from a finite set, existing arcing algorithms are shown to be algorithms for finding good game strategies. An optimal game strategy finds a combined predictor that minimizes the maximum of the error over the training set. A bound on the generalization error for the combined predictors in terms of their maximum error is proven that is sharper than bounds to date. Arcing algorithms are described that converge to the optimal strategy. Schapire et.al. [1997] offered an explanation of why Adaboost works in terms of its ability to reduce the margin. Comparing Adaboost to our optimal ar...

Citations

1714	Schapire R: A decision-theoretic generalization of online learning and an application to boosting - Freund - 1997
1325	Experiments with a new boosting algorithm - Freund, Schapire - 1996
449	Kohavi R: An empirical comparison of voting classification algorithms - Bauer - 1999
232	Arcing classifiers - Breiman - 1998
81	Boosting decision trees - Drucker, H, et al. - 1996
1	Arcing Classifiers, Technical Report 460, Statistics Department, University of California (available at www.stat.berkeley.edu) Breiman, L. [1996b] Bagging predictors - Breiman, Variance - 1997
1	Bounding Learning Algorithms, (available at http://www.research.att.com/~yoav look under "publications - Freund - 1995