Results 1 
4 of
4
Second Order Derivatives for Network Pruning: Optimal Brain Surgeon
 Advances in Neural Information Processing Systems 5
, 1993
"... We investigate the use of information from all second order derivatives of the error function to perform network pruning (i.e., removing unimportant weights from a trained network) in order to improve generalization and increase the speed of further training. ..."
Abstract

Cited by 170 (2 self)
 Add to MetaCart
We investigate the use of information from all second order derivatives of the error function to perform network pruning (i.e., removing unimportant weights from a trained network) in order to improve generalization and increase the speed of further training.
An iterative pruning algorithm for feedforward neural networks
 IEEE Trans. Neural. Networks
, 1997
"... Abstract — The problem of determining the proper size of an artificial neural network is recognized to be crucial, especially for its practical implications in such important issues as learning and generalization. One popular approach tackling this problem is commonly known as pruning and consists o ..."
Abstract

Cited by 28 (0 self)
 Add to MetaCart
Abstract — The problem of determining the proper size of an artificial neural network is recognized to be crucial, especially for its practical implications in such important issues as learning and generalization. One popular approach tackling this problem is commonly known as pruning and consists of training a larger than necessary network and then removing unnecessary weights/nodes. In this paper, a new pruning method is developed, based on the idea of iteratively eliminating units and adjusting the remaining weights in such a way that the network performance does not worsen over the entire training set. The pruning problem is formulated in terms of solving a system of linear equations, and a very efficient conjugate gradient algorithm is used for solving it, in the leastsquares sense. The algorithm also provides a simple criterion for choosing the units to be removed, which has proved to work well in practice. The results obtained over various test problems demonstrate the effectiveness of the proposed approach. Index Terms — Feedforward neural networks, generalization, hidden neurons, iterative methods, leastsquares methods, network pruning, pattern recognition, structure simplification. I.
What's Wrong with A Cascaded Correlation Learning Network: A Projection Pursuit Learning Perspective
"... Cascaded correlation is a popular supervised learning architecture that dynamically grows layers of hidden neurons of fixed nonlinear activations (e.g., sigmoids), so that the network topology (size, depth) can be efficiently determined. Similar to a cascaded correlation learning network (CCLN), a p ..."
Abstract

Cited by 7 (0 self)
 Add to MetaCart
Cascaded correlation is a popular supervised learning architecture that dynamically grows layers of hidden neurons of fixed nonlinear activations (e.g., sigmoids), so that the network topology (size, depth) can be efficiently determined. Similar to a cascaded correlation learning network (CCLN), a projection pursuit learning network (PPLN) also dynamically grows the hidden neurons. Unlike a CCLN where cascaded connections from the existing hidden units to the new candidate hidden unit are required to establish highorder nonlinearity in approximating the residual error, a PPLN approximates the highorder nonlinearity by using (more flexible) trainable nonlinear nodal activation functions. Moreover, the maximum correlation training criterion used in a CCLN results in a poorer estimate of hidden weights when compared with the minimum mean squared error criterion used in a PPLN. The CCLN is thus excluded for most regression applications where smooth interpolation of functional values are ...
Feedforward Neural Network Design with Tridiagonal Symmetry Constraints
, 1999
"... This paper introduces a pruning algorithm with tridiagonal symmetry constraints for feedforward neural network design. The algorithm uses a reflection transform applied to the inputhidden weight matrix in order to reduce it to its tridiagonal form. The designed FANN structures obtained by apply ..."
Abstract

Cited by 1 (1 self)
 Add to MetaCart
This paper introduces a pruning algorithm with tridiagonal symmetry constraints for feedforward neural network design. The algorithm uses a reflection transform applied to the inputhidden weight matrix in order to reduce it to its tridiagonal form. The designed FANN structures obtained by applying the proposed algorithm are compact and symmetrical. Therefore, they are well suited for efficient hardware and software implementations. Moreover, the number of the FANN parameters is reduced without a significant loss in performance. We illustrate the complexity and performance of the proposed algorithm by applying it as a solution to a nonlinear regression problem. We also compare the results of our proposed algorithm with those of the Optimal Brain Damage algorithm. EDICS: SP 6.1.5 This work was supported by the Natural Sciences and Engineering Research Council of Canada under contract #06P0187668. 1 Introduction Feedforward neural network (FANN) design has lately attracted...