Abstract

We review accuracy estimation methods and compare the two most common methods: crossvalidation and bootstrap. Recent experimental results on artificial data and theoretical results in restricted settings have shown that for selecting a good classifier from a set of classifiers (model selection), ten-fold cross-validation may be better than the more expensive leaveone -out cross-validation. We report on a largescale experiment---over half a million runs of C4.5 and a Naive-Bayes algorithm---to estimate the effects of different parameters on these algorithms on real-world datasets. For crossvalidation, we vary the number of folds and whether the folds are stratified or not; for bootstrap, we vary the number of bootstrap samples. Our results indicate that for real-word datasets similar to ours, the best method to use for model selection is ten-fold stratified cross validation, even if computation power allows using more folds. 1 Introduction It can not be emphasized enough that no claim ...

Citations

3144	Classification and Regression Trees - Breiman, Friedman, et al. - 1984
1785	An Introduction to the Bootstrap - Efron, Tibshirani - 1993
662	Repository of machine learning databases - Murphy, Aha - 1994
655	C4.5: Programs for - Quinlan - 1993
463	Stacked generalization - Wolpert - 1992
285	An analysis of Bayesian classifiers - Langley, Iba, et al. - 1992
173	Classi cation and Regression Trees - Breiman, Friedman, et al. - 1984
150	Estimating the error rate of a prediction rule: improvement on cross-validation - EFRON - 1983
140	A Conservation Law for Generalization Performance - Schaffer - 1994
90	MLC++: A machine learning library in C - Kohavi, John, et al. - 1994
45	Submodel selection and evaluation in regression. the x-random case - Breiman, L, et al. - 1992
37	The relationship between PAC, the statistical physics framework, the Bayesian framework, and the VC framework - Wolpert - 1994
18	Estimating the accuracy of learned concepts - Bailey, C - 1993
16	Off-training set error and a priori distinctions between learning algorithms - Wolpert - 1995
15	An analysis of bayesian classi ers - Langley, Iba, et al. - 1992
14	Decision tree pruning : Biased or optimal - Weiss, Indurkhya - 1994
13	A conservation law for generalization performance - er, C - 1994
6	Small sample error rate estimation for k-nearest neighbor classi ers - Weiss - 1991
3	Linear model selection via crossvalidation - Shao - 1993
2	Bootstrap techniques for error estimation", IEEE transactions on pattern analysis and machine intelligence PAMI-9(5 - Jain, Dubes, et al. - 1987
1	Bootstrap techniques for error estimation", IEEE transactions on pattern analysis and machine intelligence PAMI-9(5 - Jain, Dubes, et al. - 1987