Flat Minimum Search Finds Simple Nets

Flat Minimum Search Finds Simple Nets (1994)

Cached

Download Links

by Sepp Hochreiter , Jürgen Schmidhuber

Citations:

BibTeX

@TECHREPORT{Hochreiter94flatminimum,
    author = {Sepp Hochreiter and Jürgen Schmidhuber},
    title = {Flat Minimum Search Finds Simple Nets},
    institution = {},
    year = {1994}
}

Bookmark

OpenURL

Abstract

We present a new algorithm for finding low complexity neural networks with high generalization capability. The algorithm searches for a "flat" minimum of the error function. A flat minimum is a large connected region in weight-space where the error remains approximately constant. An MDL-based argument shows that flat minima correspond to low expected overfitting. Although our algorithm requires the computation of second order derivatives, it has backprop's order of complexity. Automatically, it effectively prunes units, weights, and input lines. Various experiments with feedforward and recurrent nets are described. In an application to stock market prediction, flat minimum search outperforms (1) conventional backprop, (2) weight decay, (3) "optimal brain surgeon" / "optimal brain damage".

Citations

969	Generalized Additive Models - Hastie, Tibshirani - 1990
891	Modeling by shortest data description - Rissanen - 1978
559	Cross-validatory choice and assessment of statistical predictions - STONE - 1974
375	Optimum brain damage - Cun, Denker, et al. - 1990
346	A practical Bayesian framework for back-propagation networks - MacKay - 1992
332	Smoothing noisy data with spline functions - CRAVEN, WAHBA - 1979
201	Generalized Cross-Validation as a Method for Choosing a Good Ridge Parameter - Golub, Heath, et al. - 1979
190	Spline Smoothing and Nonparametric Regression - Eubank - 1988
159	The effective number of parameters: an analysis of generalization and regularization in nonlinear learning systems”, in eds - Moody - 1992
148	Second order derivatives for network pruning: Optimal brain surgeon - Hassibi, Stork - 1993
125	Generalization by weight–elimination with applications to forecasting - Weigend, Rumelhart, et al. - 1991
120	Learning in artificial neural networks: A statistical perspective - White - 1989
113	Simplifying neural networks by soft weight-sharing - Nowlan, Hinton - 1992
111	A technique for trimming the fat from a network via relevance assessment - Mozer, Smolensky, et al. - 1988
76	Statistical predictor identification - Akaike - 1970
73	A Simple Weight Decay Can Improve Generalization - Krogh, Hertz - 1995
54	Fast exact multiplication by the Hessian - Pearlmutter - 1994
52	A Comparison of Biases For Minimal Network Construction with Back-Propagation - Hanson, Pratt - 1989
49	A minimum description length framework for unsupervised learning. Unpublished doctoral dissertation - Zemel - 1993
46	A statistical approach to learning and generalization in layered neural networks - Esther, Tishby, et al. - 1990
45	Principles of risk minimization for learning theory - Vapnik - 1992
39	Statistical theory of learning curves under entropic loss - Amari, Murata - 1993
37	The relationship between PAC, the statistical physics framework, the Bayesian framework, and the VC framework - Wolpert - 1994
34	On learning how to learn learning strategies - Schmidhuber - 1994
32	Structural risk minimization for character recognition', this volume - Guyon, Vapnik, et al. - 1992
29	A mathematical theory of communication (parts - Shannon - 1948
28	Architecture Selection Strategies for Neural Networks: Application to Bond Rating Prediction - Moody, Utans - 1995
27	The recurrent cascade-correlation learning algorithm - Fahlman - 1991
26	Fast pruning using principal components - Levin, Leen, et al. - 1994
26	Fast learning in multi--resolution hierarchies - Moody - 1989
26	Stock performance modeling using neural networks: A comparative study with regression models - Refenes, Zapranis, et al. - 1994
25	Noise injection into inputs in back-propagation learning - Matsuoka - 1992
18	Maximally fault tolerant neural networks - Neti, Schneider, et al. - 1992
17	Discovering problem solutions with low Kolmogorov complexity and high generalization capability. Technical report FKI-194-94, Fakultät für Informatik, Technische Universität - Schmidhuber - 1994
15	Optimal stopping and effective machine complexity in learning - Wang, Venkatesh - 1994
13	An information theoretic measure for classification - WALLACE, BOULTON - 1968
13	Bayesian backpropagation over i-o functions rather than weights - Wolpert - 1994
12	Data analysis, including statistics - Mosteller, Tukey - 1968
12	Chaitin-Kolmogorov complexity and generalization in neural networks - Pearlmutter, Rosenfeld - 1991
12	Bayesian regularisation and pruning using a laplace prior - Williams - 1995
10	Curvature-driven smoothing in backpropagation neural networks - Bishop - 1990
10	Keeping Neural Networks Simple - Hinton, Camp - 1993
7	Operational Fault Tolerance of CMAC Networks - Carter, Rudolph, et al. - 1990
7	Exact calculation of the product of the Hessian matrix of feed-forward network error functions and a vector in O(N) time - M��ller - 1993
7	Synaptic weight noise during MLP learning enhances fault-tolerance, generalisation and learning trajectory - Murray, Edwards - 1993
6	On the theory of generalization and self-structuring in linearly weighted connectionist networks - Holden - 1993
5	Simplifying nets by discovering flat minima - Hochreiter, Schmidhuber - 1995
5	Perturbation response in feedforward networks - Minai, Williams - 1994
4	Dynamic node creation in backpropagation neural networks - Ash - 1989
3	Robustness in multilayer perceptrons - Kerlirzin, Vallet - 1993