Abstract

Although Bayesian model averaging is theoretically the optimal method for combining learned models, it has seen very little use in machine learning. In this paper we study its application to combining rule sets, and compare it with bagging and partitioning, two popular but more ad hoc alternatives. Our experiments show that, surprisingly, Bayesian model averaging's error rates are consistently higher than the other methods'. Further investigation shows this to be due to a marked tendency to overfit on the part of Bayesian model averaging, contradicting previous beliefs that it solves (or avoids) the overfitting problem.

Citations

4071	C4.5: Programs for machine learning - Quinlan - 1993
2529	C.J.E.: Uci repository of machine learning databases, http://www.ics.uci.edu/ ∼ mlearn/MLRepository.html - Blake, Keogh
1998	Bagging predictors - Breiman - 1996
1325	Experiments with a new boosting algorithm - Freund, Schapire - 1996
787	Field Theory - Wit, Smith - 1986
507	An Introduction to Computational Learning Theory - Kearns, Vazirani - 1994
496	Statistical Language Learning - Charniak - 1993
449	Kohavi R: An empirical comparison of voting classification algorithms - Bauer - 1999
449	Probabilistic inference using markov chain monte carlo methods - Neal - 1971
426	D.: Markov chain Monte Carlo in practice - Gilks, Richardson, et al. - 1996
358	TM: Machine Learning - Mitchell - 1997
251	Bagging, boosting, and C4.5 - Quinlan - 1996
155	Efficient approximations for the marginal likelihood of Bayesian networks with hidden variables - Chickering, Heckerman - 1997
131	Error-correcting output coding corrects bias and variance - Kong, Dietterich - 1995
125	The Estimation of Probability: An Essay on Modern Bayesian Methods - Good - 1965
99	T.: Pruning adaptive boosting - Margineantu, Dietterich - 1997
91	variance and arcing classifiers - Breiman, “Bias - 1996
80	An empirical evaluation of bagging and boosting - Maclin, Opitz - 1997
77	Unifying instance–based and rule–based induction - Domingos - 1996
65	Constructing Decision Trees in Noisy Domains - Niblett - 1987
60	Predicting Sunspots and Exchange Rates with Connectionist Networks - Weigend, Rumelhart, et al. - 1991
36	On pruning and averaging decision trees - Oliver, Hand - 1995
35	Option decision trees with majority votes - Kohavi, Kunz - 1997
29	E cient approximations for the marginal likelihood of Bayesian networks with hidden variables - Chickering, Heckerman - 1997
26	Technical trading rules and regime shifts in foreign exchange, in Advanced trading rules - LeBaron - 1998
22	Boosting and other machine learning algorithms - Drucker, Cortes, et al. - 1994
14	A theory of learning classification rules. Doctoral dissertation - Buntine - 1990
12	Using partitioning to speed up specific-to-general rule induction - Domingos - 1996
5	Classification using Bayes Averaging of Multiple, Relational Rulebased Models - Ali, Pazzani - 1995
4	Visualizing bagged decision trees - Potts - 1997