Abstract

Cross validation can be used to detect when overfitting starts during supervised training of a neural network; training is then stopped before convergence to avoid the overfitting ("early stopping"). The exact criterion used for cross validation based early stopping, however, is chosen in an ad-hoc fashion by most researchers or training is stopped interactively. To aid a more well-founded selection of the stopping criterion, 14 different automatic stopping criteria from 3 classes were evaluated empirically for their efficiency and effectiveness in 12 different classification and approximation tasks using multi layer perceptrons with RPROP training. The experiments show that on the average slower stopping criteria allow for small improvements in generalization (on the order of 4%), but cost about factor 4 longer training time. 1 Training for generalization When training a neural network, one is usually interested in obtaining a network with optimal generalization performance. Genera...

Citations

590	The cascade-correlation learning architecture - Fahlman, Lebiere - 1990
537	Neural networks and the bias/variance dilemma - Geman, Bienenstock, et al. - 1992
505	A direct adaptive method for faster back-propagation learning: the RPROP algorithm - Riedmiller, Braun - 1993
375	Optimum brain damage - Cun, Denker, et al. - 1990
205	An Empirical Study of Learning Speed in Back-Propagation Networks - Fahlman - 1988
169	Pruning algorithms ± a survey - Reed - 1993
148	Second order derivatives for network pruning: Optimal brain surgeon - Hassibi, Stork - 1993
113	Simplifying neural networks by soft weight-sharing - Nowlan, Hinton - 1992
85	PROBEN1 — A set of benchmarks and benchmarking rules for neural network training algorithms,” Fakultät für Informatik, Universität - Prechelt - 1994
73	A Simple Weight Decay Can Improve Generalization - Krogh, Hertz - 1995
51	Improving model selection by nonconvergent methods - Finnoff, Hergert, et al. - 1993
34	Generalization and parameter estimation in feedforward nets: some experiments - Morgan, Bourlard - 1990
27	Comparative bibliography of ontogenic neural networks - Fiesler - 1994
26	Fast pruning using principal components - Levin, Leen, et al. - 1994
15	Optimal stopping and effective machine complexity in learning - Wang, Venkatesh - 1994
14	Temporal evolution of generalization during learning in linear networks - Baldi, Chauvin - 1991
3	Comparative bibliography ofontogenic neural networks - Fiesler - 1994
2	Optimal stopping and e ective machine complexity in learning - Wang, Venkatesh, et al. - 1994