Abstract

This is the first of two papers that use off-training set (OTS) error to investigate the assumption -free relationship between learning algorithms. This first paper discusses the senses in which there are no a priori distinctions between learning algorithms. (The second paper discusses the senses in which there are such distinctions.) In this first paper it is shown, loosely speaking, that for any two algorithms A and B, there are "as many" targets (or priors over targets) for which A has lower expected OTS error than B as vice-versa, for loss functions like zero-one loss. In particular, this is true if A is cross-validation and B is "anti-cross-validation" (choose the learning algorithm with largest cross-validation error). This paper ends with a discussion of the implications of these results for computational learning theory. It is shown that one can not say: if empirical misclassification rate is low; the Vapnik-Chervonenkis dimension of your generalizer is small; and the trainin...

Citations

3340	Pattern Classification and Scene Analysis - Duda, Hart - 1973
917	Statistical decision theory and Bayesian analysis - Berger - 1985
787	Field Theory - Wit, Smith - 1986
687	Estimation of dependences based on empirical data - Vapnik - 1982
554	The strenght of weak learnability - Schapire - 1990
517	Generalization as search - Mitchell - 1982
332	Computer systems that learn - Weiss, Kulikowski - 1991
194	Probalistic interpretation of feedforward classification network outputs, with relationships to statistical pattern recognition - Bridle - 1990
140	A Conservation Law for Generalization Performance - Schaffer - 1994
116	Machine Learning: A Theoretical Approach - Natarajan - 1991
116	Overfitting avoidance as bias - Schaffer - 1993
97	On the mean accuracy of statistical pattern recognizers - Hughes
95	Computational learning theory - Anthony, Biggs - 1992
91	Improving performance in neural networks using a boosting algorithm - Drucker, Schapire, et al. - 1993
86	Pattern recognition: human and mechanical - Watanabe - 1985
70	Improving Regression Estimation: Averaging methods for variance reduction with extensions to general convex measure optimization - Perrone - 1993
55	On the Connection Between In-sample Testing and Generalization Error - Wolpert - 1992
52	Exploring the decision forest: An empirical investigation of Occam's razor in decision tree induction - Murphy, Paxzani - 1994
48	Ensemble learning - Dietterich - 2002
44	Ockham’s razor and Bayesian analysis - Jefferys, Berger - 1992
37	The relationship between PAC, the statistical physics framework, the Bayesian framework, and the VC framework - Wolpert - 1994
30	On overfitting avoidance as bias - Wolpert - 1993
15	Local algorithms for pattern recognition and dependencies estimation - Vapnik, Bottou - 1993
6	Cross-validation estimates integrated mean squared error - Plutowski - 1994
5	et alia - Blumer - 1989
5	On the monotonicity of the performance of a Bayesian classifier - Waller, Jain - 1978
5	Filter likelihoods and exhaustive learning - Wolpert - 1994
4	On the Bayesian "Occam factors" argument for Occam's razor - Wolpert - 1994
2	Course notes for - Mitchell, Blum - 1994
1	et al. `Towards efficient agnostic learning - Kearns - 1992