Abstract

In comparison to symbolic differentiation and numerical differencing, the chain rule based technique of automatic differentiation is shown to evaluate partial derivatives accurately and cheaply. In particular it is demonstrated that the reverse mode of automatic differentiation yields any gradient vector at no more than five times the cost of evaluating the underlying scalar function. After developing the basic mathematics we describe several software implementations and briefly discuss the ramifications for optimization.

Citations

116	1981], Automatic Differentiation: techniques and applications - Rall
109	The complexity of partial derivatives - Baur, Strassen - 1983
35	A simple automatic derivative evaluation program - Wengert - 1964
31	Fitting dynamics to data - Thacker, Long - 1988
23	Sensitivity theory for nonlinear systems: I. Nonlinear functional analysis approach - Cacuci - 1981
21	Sensitivity theory for nonlinear systems: II. Extensions to additional classes of responses - Cacuci - 1981
19	Simultaneous computation of functions, partial derivatives, and estimates of rounding errors - Iri
19	The Calculus of Variations and Optimal Control - Leitmann - 1981
16	Automatic differentiation of computer programs - Kedem - 1980
16	A new two-constant equation of state - Peng, Robinson - 1976
14	Methods of fast automatic differentiation and applications - Iri, Kubota - 1987
12	Computational graphs and rounding error - Bauer - 1974
10	Reddien, Computation of cusp singularities for operator equations and their discretizations - Griewank, George - 1989
8	An exact penalty method with global convergence properties for nonlinear programming problems - Pillo, Grippo - 1986
8	The arithmetic of differentiation - Rall - 1986
7	Automatic differentiation and parallel processing in optimisation - Dixon - 1987
7	Numerical derivatives and nonlinear analysis - Kagiwada, Kalaba, et al. - 1986
5	Automatic computation of partial derivatives and rounding error estimates with applications to large-scale systems of nonlinear equations - Iri, Tsuchiya, et al. - 1988
4	1987]: Optimal implementation of differentiation arithmetic - Rall - 1987
2	S.M.Watt - Char, Gonnet - 1988
2	The Use of the Extended Operations Set of ADA with Automatic Differentiation and the Truncated Newton Method - Dixon, Mohseninia - 1987
2	Metody resheniia ekstremal'nykh zadach ikh primenenie v sistemakh optimizatsii - Evtushenko - 1982
2	Automatic Differentiation: How to compute the Hessian matrix - Fischer - 1987
2	Users Guide for JAKEF - Hillstrom - 1985
2	Second order Sensitivity Analysis - Jackson, McCormick - 1988
2	An efficient Algorithm for Computing Derivatives and extremal Problems" English translation of "Effektivnyi algoritm vychisleniia proizvodnykh i ekstremal'nye zaduchi", Ekonomika i matematicheskie metody - Kim, Nesterov, et al. - 1984