Abstract

Although Bayesian model averaging is theoretically the optimal method for combining learned models, it has seen very little use in machine learning. In this paper we study its application to combining rule sets, and compare it with bagging and partitioning, two popular but more ad hoc alternatives. Our experiments show that, surprisingly, Bayesian model averaging's error rates are consistently higher than the other methods'. Further investigation shows this to be due to a marked tendency to overfit on the part of Bayesian model averaging, contradicting previous beliefs that it solves (or avoids) the overfitting problem.

Citations

4072	JR: C4.5: Programs for Machine Learning - Quinlan
2528	UCI Repository of Machine Learning Databases - Blake, Keogh, et al. - 1998
1999	Bagging predictors - Breiman - 1996
1326	Schapire R: Experiments with a new boosting algorithm - Freund
786	Bayesian Theory - Bernardo, Smith - 1994
507	An Introduction to Computational Learning Theory - Kearns, Vazirani - 1994
496	Statistical Language Learning - Charniak - 1993
449	An Empirical Comparison of Voting Classification Algorithms - Bauer, Kohavi - 1999
448	Probabilistic inference using Markov chain Monte Carlo methods - Neal - 1993
426	D.: Markov chain Monte Carlo in practice - Gilks, Richardson, et al. - 1996
358	Machine learning - Mitchell - 1997
251	Bagging, boosting, and C4.5 - Quinlan - 1996
155	Efficient approximations for the marginal likelihood of Bayesian networks with hidden variables - Chickering, Heckerman - 1997
131	Error-correcting output coding corrects bias and variance - Kong, Dietterich - 1995
125	The Estimation of Probabilities. An Essay on Modern Bayesian Methods - Good - 1965
99	T.: Pruning adaptive boosting - Margineantu, Dietterich - 1997
91	variance and arcing classifiers - Breiman, “Bias - 1996
80	An empirical evaluation of bagging and boosting - Maclin, Opitz - 1997
77	Unifying instance–based and rule–based induction - Domingos - 1996
65	Constructing Decision Trees in Noisy Domains - Niblett - 1987
60	Predicting Sunspots and Exchange Rates with Connectionist Networks - Weigend, Rumelhart, et al. - 1991
36	On pruning and averaging decision trees - Oliver, Hand - 1995
35	Option decision trees with majority votes - Kohavi, Kunz - 1997
29	E cient approximations for the marginal likelihood of Bayesian networks with hidden variables - Chickering, Heckerman - 1997
26	Technical trading rules and regime shifts in foreign exchange, in Advanced trading rules - LeBaron - 1998
22	Boosting and other machine learning algorithms - Drucker, Cortes, et al. - 1994
14	A theory of learning classification rules. Doctoral dissertation - Buntine - 1990
12	Using partitioning to speed up specific-to-general rule induction - Domingos - 1996
5	Classification using Bayes Averaging of Multiple, Relational Rulebased Models - Ali, Pazzani - 1995
4	Visualizing bagged decision trees - Potts - 1997