e applied. In summary, AD is more accurate than numerical differentiation and requires fewer resources and is more generally applicable than symbolic differentiation. The simplest type of algorithmic definition of a function is a code list , which is similar to the segment of computer code for the evaluation of an expression (i.e., a formula). For illustration, consider the function defined by the formula f(x; y) = (xy + sin x + 4)(3y 2 + 6): An equivalent algorithmic definition of this function by a code list is t 1 = x; t 6 = t 5 + 4; t 2 = y; t 7 = t 2 2 ; t 3 = t 1 t 2 ; t 8 = 3t 7 ; t 4 = s

In comparison to symbolic differentiation and numerical differencing, the chain rule based technique of automatic differentiation is shown to evaluate partial derivatives accurately and cheaply. In particular it is demonstrated that the reverse mode of automatic differentiation yields any gradient vector at no more than five times the cost of evaluating the underlying scalar function. After developing the basic mathematics we describe several software implementations and briefly discuss the ramifications for optimization.

. Numerical solution of a singular unconstrained minimization problem is divided into two steps. Firstly, we consider a continuous analogue of the steepest gradient descent, which leads to a system of Volterra integral equations, and obtain convergence rate estimates. Secondly, the system is solved by an approximate-iterative method. An estimate of the total absolute error is given as a sum of the inherent error, the error of the numerical method, and the round-o error. The estimate enables one to determine parameters of the computational process and to solve the initial problem with pre-assigned accuracy. Key words. Unconstrained minimization, singular problems, Volterra integral equation, rate of convergence, error estimate. AMS(MOS) subject classications. 65K05, 65R20, 65G05 1. Introduction. We consider the following unconstrained minimization problem: nd x 2 R n , for which (1) f f(x ) = min x2R n f(x); where R n is the n-dimensional real Euclidean space. We a...

We discuss the complexity of evaluating certain collections of monic multilinear polynomials using operations in {×, +}. The functions we consider, which are defined on paths in directed acyclic graphs (DAGs), represent derivative computations that are based on the chain rule and have direct applications in high-performance scientific computing. Our main results concern functions derived from single-source, single-sink DAGs whose maximal paths all have length three. We derive tight, exact lower bounds for the numbers of multiplications, additions, and total arithmetic operations needed. Moreover, we show that, given such a DAG, an arithmetic circuit (or straight-line program) of minimum size that evaluates J (G) can be constructed in polynomial time. In contrast, we show the (perhaps surprising) result that the problem of finding a circuit of minimum size for a given DAG becomes NP-hard, even for the restricted class of DAGs considered in this paper, when some subset of the arcs may be labeled with the multiplicative identity 1 rather than an indeterminate. 1