Abstract

methods of model selection, and how do they work? Which methods perform better than others, and in what circumstances? These questions rest on a number of key concepts in a relatively underdeveloped field. The aim of this essay is to explain some background concepts, highlight some of the results in this special issue, and to add my own. The standard methods of model selection include classical hypothesis testing, maximum likelihood, Bayes method, minimum description length, cross-validation and Akaike’s information criterion. They all provide an implementation of Occam’s razor, in which parsimony or simplicity is balanced against goodness-of-fit. These methods primarily take account of the sampling errors in parameter estimation, although their relative success at this task depends on the circumstances. However, the aim of model selection should also include the ability of a model to generalize to predictions in a different domain. Errors of extrapolation, or generalization, are different from errors of parameter estimation. So, it seems that simplicity and parsimony may be an additional factor in managing these errors, in which case the standard methods of model selection are incomplete implementations of Occam’s razor. 1. WHAT IS MODEL SELECTION? William of Ockham (1285- 1347/49) will always be remembered for his famous postulations of Ockham’s razor (also spelled ‘Occam’), which states that entities are not to be multiplied beyond necessity. In a similar vein, Sir Isaac Newton’s first rule of hypothesizing instructs us that we are to admit no more causes of natural things than such as are both true and sufficient to explain their appearances. While they This paper is derived from a presentation at the Methods of Model Selection symposium at Indiana University

Citations

1581	Estimating the dimension of a model - SCHWARZ - 1978
1310	The Structure of Scientific Revolutions - Kuhn - 1962
1131	A new look at the statistical model identification - Akaike - 1974
840	Information theory and an extension of the maximum likelihood principle - Akaike - 1973
809	On information and sufficiency - Kullback, Leibler - 1951
650	The logic of scientific discovery - Popper - 1959
537	Neural networks and the bias/variance dilemma - Geman, Bienenstock, et al. - 1992
378	Mathematical Methods of Statistics - Cramér - 1946
194	On the psychology of prediction - Kahneman, Tversky - 1973
124	Model selection and Akaike’s Information Criterion (AIC): the general theory and its analytical extensions. Psychometrika 52:345–370 - Bozdogan - 1987
104	How the laws of physics lie - Cartwright - 1983
103	The lack of a priori distinctions between learning algorithms - Wolpert - 1996
79	Belief in the law of small numbers - Tversky, Kahneman - 1971
61	Akaike Information Criterion Statistics - Sakamoto, Ishiguro, et al. - 1999
57	1994]: ‘How to Tell When Simpler, More Unified or Less Ad Hoc Theories - Forster, Sober
44	On entropy maximization principle - Akaike - 1977
39	Minimodularity and the perception of layout - Bruno, Cutting - 1988
37	The importance of complexity in model selection - Myung - 2000
33	Bayesian model selection and model averaging - Wasserman
32	Model Selection Based on Minimum Description Length, submitted to - Grünwald - 1999
28	Akaike’s information criterion and recent developments in information complexity - Bozdogan - 2000
18	Cross-validation methods - Browne - 2000
15	forthcoming): “The New Science of Simplicity - Forster - 1997
13	Prediction and entropy - Akaike - 1985
10	Mathematical methods of statistics - Cramr - 1946
10	Statistical tests for comparing possibly misspecified and non nested models - Golden - 2000
9	Ambiguity in perception and experimentation - Massaro - 1988
6	1999]: ‘Model Selection in Science: The Problem of Language Variance’, British Journal for the Philosophy of Science, 50, pp. 83–102. Downloaded from http://bjps.oxfordjournals.org at Serial Record on April 10, 2010 not Bust! Why Simplicity is no Problem - Forster
5	Bayes or bust: the problem of simplicity for a probabilistic approach to confirmation. British Journal for the Philosophy of Science 46 - Forster - 1995
5	Yi-Ming Wang (2000): “Model Comparisons and Model Selection Based on Generalization Criterion Methodology - Busemeyer
5	ed.) William Whewell’s Theory of Scientific Method - Butts - 1968
3	The Curve Fitting Problem: A Bayesian Approach - Bandyopadhyay, Boik, et al. - 1996
3	Accuracy, scope, and flexibility of models - Cutting - 2000
2	Causal Laws and Effective Strategies," Nous - Cartwright - 1978
2	Unification, explanation, and the composition of causes in newtonian mechanics - Forster - 1988
2	Leibler (1951): "On Information and Sufficiency - Kullback, A
1	Santen (1999): "How to tell whether a model is testable: The example of discrete -state selective-influence models - Bamber, van
1	Yi-Min Wang (1999): "Model comparisons and model selections based on generalization test methodology - Busemeyer
1	Model Selection based on Minimum Description Length - Gr��nwald - 1999
1	Variation and the Accuracy of Predictions." British Journal for the Philosophy of Science 48 - Kruse - 1997
1	Santen (2000). How to assess a model’s testability and identifiability - Bamber, van
1	Variation and the accuracy of predictions - Kruse - 1997