Abstract

In this paper we describe the use of Rademacher penalization for model selection. As in Vapnik's Guaranteed Risk Minimization (GRM), Rademacher penalization attemps to balance the complexity of the model with its t to the data by minimizing the sum of the training error and a penalty term, which is an upper bound on the absolute dierence between the training error and the generalization error. However, while the GRM penalty is universal, the computation of the Rademacher penalty is data driven which means that it depends on the distribution of the data and hence one can expect better performance for particular instances of learning problems. We present experimental evidence that shows that Rademacher penalization can be used as an eective method of model selection in learning problems. In particular wehave shown that for the intervals model selection problem, Rademacher penalization outperforms GRM and cross validation (CV) over a wide range of sample sizes. Our experiments also sho...

Citations

6694	Statistical Learning Theory - Vapnik - 1998
182	Computational limitations on learning from examples - Pitt, Valiant - 1988
182	Weak Convergence and Empirical Processes with Applications to Statistics - Vaart, Wellner - 1996
101	An Experimental and Theoretical Comparison of Model Selection Methods - Kearns, Mansour, et al. - 1999
54	Rademacher penalties and structural risk minimization - Koltchinskii - 2001
38	Asymptotics for and against cross-validation - Stone - 1977
32	Training a 3-node neural net is NP-complete - Blum, Rivest - 1989
29	Cross-validatory choice and assesment of statistical predictions - Stone - 1974
16	Toward Ecient Agnostic Learning - Kearns, Schapire, et al. - 1992
5	Statistical Learning Control of Uncertain Systems: It is better than it seems - Koltchinskii, Abdallah, et al. - 1999
4	Training a sigmoidal node is hard - Hush - 1999