## Diagrammatic Derivation of Gradient Algorithms for Neural Networks (1994)

Venue: | in Neural Computation |

Citations: | 15 - 1 self |

### BibTeX

@ARTICLE{Wan94diagrammaticderivation,

author = {Eric A. Wan and Françoise Beaufays},

title = {Diagrammatic Derivation of Gradient Algorithms for Neural Networks},

journal = {in Neural Computation},

year = {1994},

volume = {8},

pages = {182--201}

}

### OpenURL

### Abstract

Deriving gradient algorithms for time-dependent neural network structures typically requires numerous chain rule expansions, diligent bookkeeping, and careful manipulation of terms. In this paper, we show how to use the principle of Network Reciprocity to derive such algorithms via a set of simple block diagram manipulation rules. The approach provides a common framework to derive popular algorithms including backpropagation and backpropagation-through-time without a single chain rule expansion. Additional examples are provided for a variety of complicated architectures to illustrate both the generality and the simplicity of the approach. 1 Introduction Deriving the appropriate gradient descent algorithm for a new network architecture or system configuration normally involves brute force derivative calculations. For example, the celebrated backpropagation algorithm for training feedforward neural networks was derived by repeatedly applying chain rule expansions backward through the ne...