Abstract

This is the first of two papers that use off-training set (OTS) error to investigate the assumption -free relationship between learning algorithms. This first paper discusses the senses in which there are no a priori distinctions between learning algorithms. (The second paper discusses the senses in which there are such distinctions.) In this first paper it is shown, loosely speaking, that for any two algorithms A and B, there are "as many" targets (or priors over targets) for which A has lower expected OTS error than B as vice-versa, for loss functions like zero-one loss. In particular, this is true if A is cross-validation and B is "anti-cross-validation" (choose the learning algorithm with largest cross-validation error). This paper ends with a discussion of the implications of these results for computational learning theory. It is shown that one can not say: if empirical misclassification rate is low; the Vapnik-Chervonenkis dimension of your generalizer is small; and the trainin...

Citations

3913	Pattern Classification and Scene Analysis - Duda, Hart - 1973
1233	Statistical Decision Theory and Bayesian Analysis - Berger - 1985
1035	Bayesian Theory - Bernardo, Smith - 1994
803	Estimation of Dependences Based on Empirical Data - Vapnik - 1982
667	The Strength of Weak Learnability - Schapire - 1990
553	Generalization as Search - Mitchell - 1982
365	Computer systems that learn - Weiss, Kulikowski - 1991
227	Probabilistic interpretation of feedforward classification network outputs, with relationships to statistical pattern recognition - Bridle - 1989
148	On the mean accuracy of statistical pattern recognizers - Hughes - 1968
144	A conservation law for generalization performance - Schaffer - 1994
120	Machine Learning: A Theoretical Approach - Natarajan - 1992
119	Overfitting avoidance as bias - Schaffer - 1993
105	Computational Learning Theory - Anthomy, Biggs - 1992
99	Pattern Recognition: Human and Mechanical - WATANABE - 1985
95	Improving performance in neural networks using a boosting algorithm - Drucker, Schapire, et al. - 1993
82	Improving Regression Estimation: Averaging Methods for Variance Reduction with Extension to General Convex Measure Optimization - Perrone - 1993
63	Ockham’s razor and Bayesian analysis - Jefferys, Berger - 1992
60	On the connection between in-sample testing and generalization error. Complex Systems 6:47–94 - Wolpert - 1992
57	Exploring the decision forest: an empirical investigation of Occam’s razor in decision tree induction - Murphy, Pazzani - 1994
53	Ensemble learning - Dietterich - 2003
40	The relationship between PAC, the statistical physics framework, the Bayesian framework, and the VC framework. In The mathematics of generalization, edited by D. Wo1pert - Wolpert - 1995
33	On overfitting avoidance as bias - Wolpert
17	Local algorithms for pattern recognition and dependencies estimation, Neural Computation 5(6):893–909 - Vapnik, Bottou - 1993
6	Cross-validation estimates integrated mean squared error - Plutowski - 1994
5	et alia - Blumer - 1989
5	On the monotonicity of the performance of Bayesian classifiers - Waller, Jain - 1978
5	Filter likelihoods and exhaustive learning - Wolpert - 1994
4	On the Bayesian "Occam factors" argument for Occam's razor - Wolpert - 1994
2	Course notes for - Mitchell, Blum - 1994
1	et al. `Towards efficient agnostic learning - Kearns - 1992