Abstract

This paper discusses the intimate relationships between the supervised learning frameworks mentioned in the title. In particular, it shows how all those frameworks can be viewed as particular instances of a single overarching formalism. In doing this many commonly misunderstood aspects of those frameworks are explored. In addition the strengths and weaknesses of those frameworks are compared, and some novel frameworks are suggested (resulting, for example, in a "correction" to the familiar bias-plus-variance formula).

Citations

3339	Pattern Classification and Scene Analysis - Duda, Hart - 1973
1515	A theory of the learnable - Valiant - 1984
917	Statistical decision theory and Bayesian analysis - Berger - 1985
687	Estimation of dependences based on empirical data - Vapnik - 1982
565	Learnability and the Vapnik Chervonenkis dimension - Blumer, Ehrenfeucht, et al. - 1989
554	The strenght of weak learnability - Schapire - 1990
537	Neural networks and the bias/variance dilemma - Geman, Bienenstock, et al. - 1992
510	Statistical Analysis of Finite Mixture Distributions - Titterington, Smith, et al. - 1985
332	Computer systems that learn - Weiss, Kulikowski - 1991
327	Decision theoretic generalizations of the PAC model for neural net and other learning applications - Haussler - 1992
296	What size net gives valid generalization - Baum, Haussler - 1989
236	Information-Based Objective Functions for Active Data - MacKay - 1992
194	Probalistic interpretation of feedforward classification network outputs, with relationships to statistical pattern recognition - Bridle - 1990
190	Spline Smoothing and Nonparametric Regression - Eubank - 1988
159	The effective number of parameters: an analysis of generalization and regularization in nonlinear learning systems”, in eds - Moody - 1992
125	Generalization by weight–elimination with applications to forecasting - Weigend, Rumelhart, et al. - 1991
119	Neural networks and related methods for classification - Ripley - 1994
116	Overfitting avoidance as bias - Schaffer - 1993
111	Bayesian back–propagation - Buntine, Weigend - 1991
98	Bounds on the Sample Complexity of Bayesian Learning Using Information Theory and the VC Dimension - Haussler, Kearns, et al. - 1991
91	Improving performance in neural networks using a boosting algorithm - Drucker, Schapire, et al. - 1993
77	A Theory of Learning Classification Rules - Buntine - 1992
73	In defense of probability - Cheeseman - 1985
70	Improving Regression Estimation: Averaging methods for variance reduction with extensions to general convex measure optimization - Perrone - 1993
67	Developments in maximum entropy data analysis - Gull - 1988
59	A comparison of GCV and GML for choosing the smoothing parameter in the generalized spline smoothing problem - Wahba - 1985
55	On the Connection Between In-sample Testing and Generalization Error - Wolpert - 1992
48	Ensemble learning - Dietterich - 2002
41	Statistical mechanics of learning from examples - Seung, Sompolinsky, et al. - 1992
30	On overfitting avoidance as bias - Wolpert - 1993
26	Generalization performance of Bayes optimal classification algorithm for learning a perceptron - Opper, Haussler - 1991
21	Priors for infinite networks - Neal - 1994
19	On the use of evidence in neural networks - Wolpert - 1993
15	Calculation of the learning curve of Bayes optimal classification algorithm for learning a perceptron with noise - Opper, Haussler - 1991
15	Local algorithms for pattern recognition and dependencies estimation - Vapnik, Bottou - 1993
13	How tight are the Vapnik-Chervonenkis bounds - Cohn, Tesauro - 1991
13	Bayesian backpropagation over i-o functions rather than weights - Wolpert - 1994
12	Reconciling Bayesian and non-Bayesian analysis - Wolpert - 1994
7	The probably approximately correct (PAC) and other learning models. Chapter 9 from Foundations of knowledge acquisition - Haussler - 1994
7	A quantitative Occam's razor - Sorkin - 1983
5	From Laplace to supernova 1987a: Bayesian inference in astrophysics - Loredo - 1990
5	Statistical physics models of supervised learning - Tishby - 1995
5	Filter likelihoods and exhaustive learning - Wolpert - 1994
4	Bayesian Interpolation," "A Practical Framework for Backpropagation Networks - MacKay - 1992
4	On the Bayesian "Occam factors" argument for Occam's razor - Wolpert - 1994
3	Generalization in feedforward neural and Boolean networks - Broeck, Kawai - 1991
2	PAB-decisions for boolean and real-valued features - Anoulova - 1992
1	Course notes and reprints for Machine Learning course 18.428, MIT - Rivest - 1989
1	Alpha, Evidence, and the entropic prior. In Maximum Entropy and Bayesian Methods, A. MOhammed-Djafari (Ed.). Kluwer Tishby N., et al - Strauss - 1993
1	How good generalizers need to be: beyond bias plus variance - Wolpert - 1994