No Unbiased Estimator of the Variance of K-Fold Cross-Validation

DMCA

No Unbiased Estimator of the Variance of K-Fold Cross-Validation (2003)

Cached

Download Links

Other Repositories/Bibliography

DBLP

by Yoshua Bengio , Yves Grandvalet

Venue:	JOURNAL OF MACHINE LEARNING RESEARCH
Citations:	60 - 1 self

BibTeX

@ARTICLE{Bengio03nounbiased,
    author = {Yoshua Bengio and Yves Grandvalet},
    title = {No Unbiased Estimator of the Variance of K-Fold Cross-Validation},
    journal = {JOURNAL OF MACHINE LEARNING RESEARCH},
    year = {2003},
    volume = {5},
    pages = {1089--1105}
}

OpenURL

Abstract

Most machine learning researchers perform quantitative experiments to estimate generalization error and compare algorithm performances. In order to draw statistically convincing conclusions, it is important to estimate the uncertainty of such estimates. This paper studies the estimation of uncertainty around the K-fold cross-validation estimator. The main theorem shows that there exists no universal unbiased estimator of the variance of K-fold cross-validation. An analysis based on the eigendecomposition of the covariance matrix of errors helps to better understand the nature of the problem and shows that naive estimators may grossly underestimate variance, as conrmed by numerical experiments.

Keyphrases

k-fold cross-validation unbiased estimator generalization error quantitative experiment naive estimator covariance matrix paper study compare algorithm performance main theorem numerical experiment k-fold cross-validation estimator convincing conclusion

DMCA