Abstract

Standard techniques for improved generalisation from neural networks include weight decay and pruning. Weight decay has a Bayesian interpretation with the decay function corresponding to a prior over weights. The method of transformation groups and maximum entropy indicates a Laplace rather than a Gaussian prior. After training, the weights then arrange themselves into two classes: (1) those with a common sensitivity to the data error (2) those failing to achieve this sensitivity and which therefore vanish. Since the critical value is determined adaptively during training, pruning---in the sense of setting weights to exact zeros---becomes a consequence of regularisation alone. The count of free parameters is also reduced automatically as weights are pruned. A comparison is made with results of MacKay using the evidence framework and a Gaussian regulariser. 1 Introduction Neural networks designed for regression or classification need to be trained using some form of stabilisation or re...

Citations

1126	Practical Optimization - Gill, Murray, et al. - 1981
882	Practical Methods of Optimization - Fletcher - 1987
593	Solution of IllPosed problems - Tikhonov, Arsenin - 1977
417	Bayesian interpolation - MacKay - 1992
375	Optimum brain damage - Cun, Denker, et al. - 1990
346	A practical Bayesian framework for backpropagation networks.Neural computation - MacKay
148	Second order derivatives for network pruning: Optimal brain surgeon - Hassibi, Stork - 1993
125	Generalization by weight–elimination with applications to forecasting - Weigend, Rumelhart, et al. - 1991
111	Bayesian back–propagation - Buntine, Weigend - 1991
73	Experiments on Learning by BackPropagation - Plaut, Nowlan, et al. - 1986
71	Hopfield; Large automatic learning, rule extraction and generalization - Denker, Schwartz, et al. - 1987
53	Rational Descriptions, Decisions and Designs - Tribus - 1969
44	A scaled conjugate gradient algorithm for fast supervised learning - Mller - 1993
34	Bayesian training of backpropagation networks by the hybrid Monte Carlo method - Neal - 1992
27	Bayesian learning via stochastic dynamics - Neal - 1993
19	On the use of evidence in neural networks - Wolpert - 1993
13	Hyperparameters: Optimise, or integrate out - MacKay - 1993
10	Curvature-driven smoothing in backpropagation neural networks - Bishop - 1990
10	A marquardt algorithm for choosing the step-size in backpropagation learning with conjugate gradients - Williams - 1991
8	Adaptive soft weight tying using gaussian mixtures - Nowlan, Hinton
7	Exact calculation of the product of the Hessian matrix of feed-forward network error functions and a vector in O(N) time - M��ller - 1993
2	probabilities - Prior - 1968
2	Thodberg. Ace of bayes : Application of neural networks with pruning - Henrik - 1993
2	Aeromagnetic compensation using neural networks - Williams - 1993
1	A Pearlmutter. Fast exact multiplication by the Hessian - Barak - 1994
1	Improved generalization and network pruning using adaptive Laplace regularization - Williams - 1993