Abstract

Occam's razor has been the subject of much controversy. This paper argues that this is partly because it has been interpreted in two quite different ways, the first of which (simplicity is a goal in itself) is essentially correct, while the second (simplicity leads to greater accuracy) is not. The paper reviews the large variety of theoretical arguments and empirical evidence for and against the "second razor," and concludes that the balance is strongly against it. In particular, it builds on the case of (Schaffer, 1993) and (Webb, 1996) by considering additional theoretical arguments and recent empirical evidence that the second razor fails in most domains. A version of the first razor more appropriate to KDD is proposed, and we argue that continuing to apply the second razor risks causing significant opportunities to be missed. 1 Occam's Two Razors William of Occam's famous razor states that "Nunquam ponenda est pluralitas sin necesitate," which, approximately translated, means "En...

Citations

6694	Statistical Learning Theory - Vapnik - 1998
6514	Elements of Information Theory - Cover, Thomas - 1991
3909	Neural networks for pattern recognition - Bishop - 1995
787	Field Theory - Wit, Smith - 1986
486	Pazzani M: On the optimality of the simple Bayesian classifier under zero-one loss - Domingos - 1997
385	Very simple classification rules perform well on most commonly used datasets - Holte - 1993
274	Inferring decision trees using the minimum description length principle. Information and Computation 80.227–248 - QUINLAN, RIVEST - 1989
270	An information measure for classification - Wallace, Boulton - 1968
251	Bagging, boosting, and C4.5 - Quinlan - 1996
207	Quantifying inductive bias: AI learning algorithms and Valiant’s learning framework - Haussler - 1988
167	The Need for Biases in Learning Generalizations - Mitchell - 1990
166	On bias, variance, 0/1 - loss, and the curse-of-dimensionality, Data Mining and Knowledge Discovery 1 - Friedman - 1997
156	An Empirical Comparison of Pruning Methods for Decision–Tree Induction - Mingers - 1989
140	A Conservation Law for Generalization Performance - Schaffer - 1994
120	Modeling by shortest data description. Automatica 14 - Rissanen - 1978
116	Overfitting avoidance as bias - Schaffer - 1993
110	Theory refinement combining analytical and empirical methods. Artificial Intelligence 66:311--344 - Ourston, Mooney - 1994
82	Learning from Hints in Neural Networks - Abu–Mostafa - 1990
76	Cameron-Jones. Oversearching and Layered Search in Empirical Learning - Quinlan, M - 1995
65	Extracting Comprehensible Models from Trained Neural Networks - Craven - 1996
52	Exploring the decision forest: An empirical investigation of Occam's razor in decision tree induction - Murphy, Paxzani - 1994
51	Knowledge acquisition from examples via multiple models - Domingos - 1997
51	Further experimental evidence against the utility of Occam’s Razor - Webb
49	RL4: A tool for knowledge-based induction - Clearwater, Provost - 1990
46	Using qualitative models to guide induction learning - Clark, Matwin - 1993
45	Lookahead and Pathology in Decision Tree Induction - Murthy, Salzberg - 1995
38	On Finding the Most Probable Model - Cheeseman - 1990
37	A new metric-based approach to model selection - Schuurmans - 1997
31	Beyond concise and colorful: learning intelligible rules - Pazzani, Mani, et al. - 1997
29	E cient approximations for the marginal likelihood of Bayesian networks with hidden variables - Chickering, Heckerman - 1997
29	New measurements highlight the importance of redundant knowledge - Gams - 1989
27	Why does bagging work? a Bayesian account and its implications - Domingos - 1997
27	On the connection between the complexity and credibility of inferred models - Pearl - 1978
26	A Bayesian analysis of the minimum AIC procedure - AKAIKE - 1978
25	Concept simplification and predictive accuracy - Fisher, Schlimmer - 1988
24	Estimating the dimension of a model. The Annals of Statistics 6: 461–464. doi: 10.1214/aos/1176344136 Tabei Y, Kiryu H, Kin T, Asai K (2008) A fast structural multiple alignment method for long RNA sequences. BMC Bioinformatics 9: 33. doi: 10.1186/1471-21 - Schwarz - 1978
20	Learning prototypical concept descriptions - Datta, Kibler - 1995
18	Knowledge-based learning in exploratory science: Learning rules to predict rodent carcinogenicity - Lee, Buchanan, et al. - 1998
16	Overfitting explained - Cohen, Jensen - 1997
16	Lessons in neural network training: overfitting may be harder than expected - Lawrence, Giles, et al. - 1997
15	A process-oriented heuristic for model selection - Domingos - 1998
15	Adjusting for multiple comparisons in decision tree pruning - Jensen, Schmill - 1997
15	Characterizing the generalization performance of model selection strategies - Schuurmans, Ungar - 1997
11	Constructive induction using fragmentary knowledge - Donoho - 1996
11	Induction of condensed determinations - Langley - 1996
10	Ockham: Studies and Selections - Tornay - 1938
9	defense of C4.5: Notes on learning one-level decision trees - Elomaa, In - 1994
9	Decision tree grafting - Webb - 1997
8	Learning redundant rules in noisy domains - Cestnik, Bratko - 1988
7	Unifying instance-based and rulebased induction. Machine Learning 24(2):141--168 - Domingos - 1996