On-line first order backpropagation is sufficiently fast and effective for many large-scale classification problems but for very high precision mappings, batch processing may be the method of choice. This paper reviews first- and second-order optimization methods for learning in feedforward neural networks. The viewpoint is that of optimization: many methods can be cast in the language of optimization techniques, allowing the transfer to neural nets of detailed results about computational complexity and safety procedures to ensure convergence and to avoid numerical problems. The review is not intended to deliver detailed prescriptions for the most appropriate methods in specific applications, but to illustrate the main characteristics of the different methods and their mutual relations.

Recent improvements in the capabilities of complementarity solvers have led to an increased interest in using the complementarity problem framework to address practical problems arising in mathematical programming, economics, engineering, and the sciences. As a result, increasingly more difficult problems are being proposed that exceed the capabilities of even the best algorithms currently available. There is, therefore, an immediate need to improve the capabilities of complementarity solvers. This thesis addresses this need in two significant ways. First, the thesis proposes and develops a proximal perturbation strategy that enhances the robustness of Newton-based complementarity solvers. This strategy enables algorithms to reliably find solutions even for problems whose natural merit functions have strict local minima that are not solutions. Based upon this strategy, three new algorithms are proposed for solving nonlinear mixed complementarity problems that represent a significant improvement in robustness over previous algorithms. These algorithms have local Q-quadratic convergence behavior, yet depend only on a pseudo-monotonicity assumption to achieve global convergence from arbitrary starting points. Using the MCPLIB and GAMSLIB test libraries, we perform extensive computational tests that demonstrate the effectiveness of these algorithms on realistic problems. Second, the thesis extends some previously existing algorithms to solve more general problem classes. Specifically, the NE/SQP method of Pang & Gabriel (1993), the semismooth equations approach of De Luca, Facchinei & Kanz...

This thesis is concerned with algorithms and software for the solution of the Mixed Complementarity Problem, or MCP. The MCP formulation is useful for expressing systems of nonlinear inequalities and equations; the complementarity allows boundary conditions be to specified in a succinct manner. Problems of this type occur in many branches of the sciences, including mathematics, engineering, economics, operations research, and computer science. The algorithm we propose for the solution of MCP is a Newton based method containing a novel application of a nonmonotone stabilization technique previously applied to methods for solving smooth systems of equalities and for unconstrained minimization. In order to apply this technique, we have adapted and extended the path construction technique of Ralph (1994), resulting in the PATH algorithm. We present a global convergence result for the PATH algorithm that generalizes similar results obtained in the smooth case. The PATH solver is a sophisticated implementation of this algorithm that makes use of the sparse basis updating package of MINOS 5.4. Due to the widespread use of algebraic modeling languages in the practice of operations research, economics, and other fields from which complementarity problems are drawn, we have developed a complementarity facility for both the GAMS and AMPL modeling languages, as well as software interface libraries to be used in hooking up a complementarity solver as a solution subsystem. These interface libraries provide the algorithm developer with a convenient and efficient means of developing and testing an algorithm, ...

Many problems involving fluid flow can now be simulated numerically, providing a useful predictive tool for a wide range of engineering applications. Of particular interest in this thesis are computational methods for solving the problem of compressible fluid flow around aerodynamic configurations. A finite element method is presented for solving the compressible Navier-Stokes equations in two dimensions on unstructured meshes. The method is stabilized by the addition of a least-squares operator (an inexpensive simplification of the Galerkin least-squares method), leading to solutions free of spurious oscillations. Convergence to steady state is reached via a backward Euler time-stepping scheme, and the use of local time-steps allows convergence to be accelerated. The choice of both the nonlinear solver, which is employed to solve the algebraic system arising at each time-step, and the iterative method used within this solver, is fully discussed, along with an inexpensive technique for...

This paper discusses the calculation of sensitivities, or derivatives, for optimization problems involving systems governed by differential equations and other state relations. The subject is examined from the point of view of nonlinear programming, beginning with the analytical structure of the first and second derivatives associated with such problems and the relation of these derivatives to implicit differentiation and equality constrained optimization. We also outline an error analysis of the analytical formulae and compare the results with similar results for finite-difference estimates of derivatives. We then attend to an investigation of the nature of the adjoint method and the adjoint equations and their relation to directions of steepest descent. We illustrate the points discussed with an optimization problem in which the variables are the coefficients in a differential operator. This research was supportedby the National Aeronautics and Space Administrationunder NASA Contra...

Ariyawansa [2] has introduced a line search termination criterion for collinear scaling minimization algorithms. However, he verified that this criterion provides sufficient decrease only in the case of objective functions that satisfy certain strong convexity assumptions. In this paper, we report results of our attempts to relax these assumptions. We present an example of a continuously differentiable, nonconvex function with bounded level sets on which the criterion of Ariyawansa [2] does not provide sufficient decrease. This example indicates that specifying line search termination criteria for collinear scaling algorithms is still an open problem. On a more positive note, we are able to show that the class of objective functions on which the criterion of Ariyawansa [2] provides sufficient decrease can be extended to include convex and strictly pseudo-convex functions.