Abstract

This is the first of two papers that use off-training set (OTS) error to investigate the assumption -free relationship between learning algorithms. This first paper discusses the senses in which there are no a priori distinctions between learning algorithms. (The second paper discusses the senses in which there are such distinctions.) In this first paper it is shown, loosely speaking, that for any two algorithms A and B, there are "as many" targets (or priors over targets) for which A has lower expected OTS error than B as vice-versa, for loss functions like zero-one loss. In particular, this is true if A is cross-validation and B is "anti-cross-validation" (choose the learning algorithm with largest cross-validation error). This paper ends with a discussion of the implications of these results for computational learning theory. It is shown that one can not say: if empirical misclassification rate is low; the Vapnik-Chervonenkis dimension of your generalizer is small; and the trainin...

Citations

3768	Pattern Classification and Scene Analysis - Duda, Hart - 1973
1172	Statistical Decision Theory and Bayesian Analysis - BERGER - 1985
974	Bayesian Theory - Bernardo, Smith - 1994
763	Estimation of Dependencies Based on Empirical Data - Vapnik - 1979
633	The strength of weak learnability - Schapire - 1990
541	Generalization as search - Mitchell - 1982
350	Computer systems that learn - Weiss, Kulikowski - 1990
220	Probabilistic interpretation of feedforward classi network outputs, relationships to statistical pattern recognition - Bridle - 1990
145	Conservation law for generalization performance - Schaffer - 1994
117	On the mean accuracy of statistical pattern recognizers - Hughes - 1968
116	Machine Learning: A Theoretical Approach - Natarajan - 1991
116	Overfitting avoidance as bias - Schaffer - 1993
101	Computational Learning Theory - Anthony, Biggs - 1992
93	Improving performance in neural networks using a boosting algorithm - Drucker, Schapire, et al. - 1993
92	Pattern Recognition: Human and Mechanical - WATANABE - 1985
78	Improving Regression Estimation: Averaging Methods for Variance Reduction with Extensions to General Convex Measure Optimization - Perrone - 1993
58	Ockham’s razor and Bayesian analysis - JeVreys, Berger - 1992
58	On the connection between in-sample testing and generalization error - Wolpert - 1992
57	Exploring the decision forest: an empirical investigation of Occam’s razor in decision tree induction, volume IV: Making Learning Systems Practical - Murphy, Pazzani - 1997
53	Ensemble Learning - Dietterich - 2002
38	The relationship between PAC, the statistical physics framework, the bayesian framework, and the VC framework - Wolpert - 1995
32	On overfitting avoidance as bias - Wolpert - 1995
16	Local algorithms for pattern recognition and dependencies estimation, Neural Computation 5(6 - Vapnik, Bottou - 1993
6	Cross-validation estimates integrated mean squared error - Plutowski - 1994
5	et alia - Blumer - 1989
5	On the monotonicity of the performance of Bayesian classifiers - Waller, Jain - 1978
5	Filter likelihoods and exhaustive learning - Wolpert - 1994
4	On the Bayesian "Occam factors" argument for Occam's razor - Wolpert - 1994
2	Course notes for - Mitchell, Blum - 1994
1	et al. `Towards efficient agnostic learning - Kearns - 1992