Abstract

An ensemble consists of a set of independently trained classi ers (such as neural networks or decision trees) whose predictions are combined when classifying novel instances. Previous research has shown that an ensemble as a whole is often more accurate than any of the single classiers in the ensemble. Bagging (Breiman 1996a) and Boosting (Freund & Schapire 1996) are two relatively new but popular methods for producing ensembles. In this paper we evaluate these methods using both neural networks and decision trees as our classi cation algorithms. Our results clearly showtwo important facts. The rst is that even though Bagging almost always produces a better classi er than any of its individual component classi ers and is relatively impervious to over tting, it does not generalize any better than a baseline neural-network ensemble method. The second is that Boosting is apowerful technique that can usually produce better ensembles than Bagging � however, it is more susceptible to noise and can quickly over t a data set.

Citations

1998	Bagging predictors - Breiman - 1996
1783	An Introduction to the Bootstrap - Tibshirani, R - 1993
1325	Experiments with a new boosting algorithm - Freund, Schapire - 1996
662	Repository of machine learning databases - Murphy, Aha - 1994
425	Neural network ensembles - Hansen, Salamon - 1990
354	Neural network ensembles, cross validation, and active learning - Krogh, Vedelsby - 1995
264	Classification and Regression - Breiman, Friedman, et al. - 1984
251	Bagging, boosting, and C4.5 - Quinlan - 1996
232	Arcing classifiers - Breiman - 1998
141	Combining forecasts: A review and annotated bibliography - Clemen - 1989
96	Generating accurate and diverse members of a neural network ensemble - Opitz, Shavlik - 1996
81	Boosting decision trees - Drucker, H, et al. - 1996
70	Improving Regression Estimation: Averaging methods for variance reduction with extensions to general convex measure optimization - Perrone - 1993
54	Synergy of clustering multiple back-propagation networks - Lincoln, Skrzypek - 1990
54	Actively searching for an effective neural-network ensemble - Opitz, Shavlik - 1996
47	Learning with ensembles: How overfitting can be useful - Sollich, Krogh - 1996
35	Combining the predictions of multiple classifiers: Using competitive learning to initialize neural networks - Maclin, Shavlik - 1995
23	Multiple networks for function learning - Alpaydin - 1993
22	Boosting and other machine learning algorithms - Drucker, Cortes, et al. - 1994