Abstract

: We give an analysis of the generalization error of cross validation in terms of two natural measures of the difficulty of the problem under consideration: the approximation rate (the accuracy to which the target function can be ideally approximated as a function of the number of hypothesis parameters), and the estimation rate (the deviation between the training and generalization errors as a function of the number of hypothesis parameters). The approximation rate captures the complexity of the target function with respect to the hypothesis model, and the estimation rate captures the extent to which the hypothesis model suffers from overfitting. Using these two measures, we give a rigorous and general bound on the error of cross validation. The bound clearly shows the tradeoffs involved with making fl --- the fraction of data saved for testing --- too large or too small. By optimizing the bound with respect to fl, we then argue (through a combination of formal analysis, plotting, and ...

Citations

816	On the uniform convergence of relative frequencies of events to their probabilities. Theory of Prob. and its - Vapnik, Chervonenkis - 1971
687	Estimation of dependences based on empirical data - Vapnik - 1982
559	Cross-validatory choice and assessment of statistical predictions - STONE - 1974
313	Universal approximation bounds for superpositions of a sigmoidal function - Barron - 1993
176	Minimum complexity density estimation - Barron, Cover - 1991
101	An Experimental and Theoretical Comparison of Model Selection Methods - Kearns, Mansour, et al. - 1999
52	Rigorous learning curve bounds from statistical mechanics - Haussler, Kearns, et al. - 1994
41	Statistical mechanics of learning from examples - Seung, Sompolinsky, et al. - 1992
39	Stochastic Complexity in Statistical Inquiry, volume 15 - Rissanen - 1989
38	Asymptotics for and against cross-validation - Stone - 1977