Abstract

The error rate of decision-tree and other classification learners can often be much reduced by bagging: learning multiple models from bootstrap samples of the database, and combining them by uniform voting. In this paper we empirically test two alternative explanations for this, both based on Bayesian learning theory: (1) bagging works because it is an approximation to the optimal procedure of Bayesian model averaging, with an appropriate implicit prior; (2) bagging works because it effectively shifts the prior to a more appropriate region of model space. All the experimental evidence contradicts the first hypothesis, and confirms the second. Bagging Bagging (Breiman 1996a) is a simple and effective way to reduce the error rate of many classification learning algorithms. For example, in the empirical study described below, it reduces the error of a decision-tree learner in 19 of 26 databases, by 4% on average. In the bagging procedure, given a training set of size s, a "bootstrap" re...

Citations

4071	C4.5: Programs for machine learning - Quinlan - 1993
2529	C.J.E.: Uci repository of machine learning databases, http://www.ics.uci.edu/ ∼ mlearn/MLRepository.html - Blake, Keogh
1998	Bagging predictors - Breiman - 1996
1325	Experiments with a new boosting algorithm - Freund, Schapire - 1996
787	Field Theory - Wit, Smith - 1986
166	On bias, variance, 0/1 - loss, and the curse-of-dimensionality, Data Mining and Knowledge Discovery 1 - Friedman - 1997
147	Constructing optimal binary decision trees is NP-complete - l, Rivest - 1976
144	Bias plus variance decomposition for zero-one loss functions - Kohavi, Wolpert - 1996
131	Error-correcting output coding corrects bias and variance - Kong, Dietterich - 1995
91	variance and arcing classifiers - Breiman, “Bias - 1996
77	A Theory of Learning Classification Rules - Buntine - 1992
51	Knowledge acquisition from examples via multiple models - Domingos - 1997
38	On Finding the Most Probable Model - Cheeseman - 1990
29	Bayesian Model Averaging - Madigan, Raftery - 1996