Results 1 
7 of
7
First and SecondOrder Methods for Learning: between Steepest Descent and Newton's Method
 Neural Computation
, 1992
"... Online first order backpropagation is sufficiently fast and effective for many largescale classification problems but for very high precision mappings, batch processing may be the method of choice. This paper reviews first and secondorder optimization methods for learning in feedforward neura ..."
Abstract

Cited by 126 (6 self)
 Add to MetaCart
Online first order backpropagation is sufficiently fast and effective for many largescale classification problems but for very high precision mappings, batch processing may be the method of choice. This paper reviews first and secondorder optimization methods for learning in feedforward neural networks. The viewpoint is that of optimization: many methods can be cast in the language of optimization techniques, allowing the transfer to neural nets of detailed results about computational complexity and safety procedures to ensure convergence and to avoid numerical problems. The review is not intended to deliver detailed prescriptions for the most appropriate methods in specific applications, but to illustrate the main characteristics of the different methods and their mutual relations.
Algorithms For Complementarity Problems And Generalized Equations
, 1995
"... Recent improvements in the capabilities of complementarity solvers have led to an increased interest in using the complementarity problem framework to address practical problems arising in mathematical programming, economics, engineering, and the sciences. As a result, increasingly more difficult pr ..."
Abstract

Cited by 41 (5 self)
 Add to MetaCart
Recent improvements in the capabilities of complementarity solvers have led to an increased interest in using the complementarity problem framework to address practical problems arising in mathematical programming, economics, engineering, and the sciences. As a result, increasingly more difficult problems are being proposed that exceed the capabilities of even the best algorithms currently available. There is, therefore, an immediate need to improve the capabilities of complementarity solvers. This thesis addresses this need in two significant ways. First, the thesis proposes and develops a proximal perturbation strategy that enhances the robustness of Newtonbased complementarity solvers. This strategy enables algorithms to reliably find solutions even for problems whose natural merit functions have strict local minima that are not solutions. Based upon this strategy, three new algorithms are proposed for solving nonlinear mixed complementarity problems that represent a significant improvement in robustness over previous algorithms. These algorithms have local Qquadratic convergence behavior, yet depend only on a pseudomonotonicity assumption to achieve global convergence from arbitrary starting points. Using the MCPLIB and GAMSLIB test libraries, we perform extensive computational tests that demonstrate the effectiveness of these algorithms on realistic problems. Second, the thesis extends some previously existing algorithms to solve more general problem classes. Specifically, the NE/SQP method of Pang & Gabriel (1993), the semismooth equations approach of De Luca, Facchinei & Kanz...
Robust Solution Of Mixed Complementarity Problems
, 1994
"... This thesis is concerned with algorithms and software for the solution of the Mixed Complementarity Problem, or MCP. The MCP formulation is useful for expressing systems of nonlinear inequalities and equations; the complementarity allows boundary conditions be to specified in a succinct manner. Prob ..."
Abstract

Cited by 9 (0 self)
 Add to MetaCart
This thesis is concerned with algorithms and software for the solution of the Mixed Complementarity Problem, or MCP. The MCP formulation is useful for expressing systems of nonlinear inequalities and equations; the complementarity allows boundary conditions be to specified in a succinct manner. Problems of this type occur in many branches of the sciences, including mathematics, engineering, economics, operations research, and computer science. The algorithm we propose for the solution of MCP is a Newton based method containing a novel application of a nonmonotone stabilization technique previously applied to methods for solving smooth systems of equalities and for unconstrained minimization. In order to apply this technique, we have adapted and extended the path construction technique of Ralph (1994), resulting in the PATH algorithm. We present a global convergence result for the PATH algorithm that generalizes similar results obtained in the smooth case. The PATH solver is a sophisticated implementation of this algorithm that makes use of the sparse basis updating package of MINOS 5.4. Due to the widespread use of algebraic modeling languages in the practice of operations research, economics, and other fields from which complementarity problems are drawn, we have developed a complementarity facility for both the GAMS and AMPL modeling languages, as well as software interface libraries to be used in hooking up a complementarity solver as a solution subsystem. These interface libraries provide the algorithm developer with a convenient and efficient means of developing and testing an algorithm, ...
A Nonlinear Programming Perspective on Sensitivity Calculations for Systems Governed By State Equations
, 1997
"... This paper discusses the calculation of sensitivities, or derivatives, for optimization problems involving systems governed by differential equations and other state relations. The subject is examined from the point of view of nonlinear programming, beginning with the analytical structure of the fir ..."
Abstract

Cited by 4 (2 self)
 Add to MetaCart
This paper discusses the calculation of sensitivities, or derivatives, for optimization problems involving systems governed by differential equations and other state relations. The subject is examined from the point of view of nonlinear programming, beginning with the analytical structure of the first and second derivatives associated with such problems and the relation of these derivatives to implicit differentiation and equality constrained optimization. We also outline an error analysis of the analytical formulae and compare the results with similar results for finitedifference estimates of derivatives. We then attend to an investigation of the nature of the adjoint method and the adjoint equations and their relation to directions of steepest descent. We illustrate the points discussed with an optimization problem in which the variables are the coefficients in a differential operator. This research was supportedby the National Aeronautics and Space Administrationunder NASA Contra...
Adaptive Stable Finite Element Methods for the Compressible NavierStokes Equations
, 1995
"... Many problems involving fluid flow can now be simulated numerically, providing a useful predictive tool for a wide range of engineering applications. Of particular interest in this thesis are computational methods for solving the problem of compressible fluid flow around aerodynamic configurations. ..."
Abstract

Cited by 4 (2 self)
 Add to MetaCart
Many problems involving fluid flow can now be simulated numerically, providing a useful predictive tool for a wide range of engineering applications. Of particular interest in this thesis are computational methods for solving the problem of compressible fluid flow around aerodynamic configurations. A finite element method is presented for solving the compressible NavierStokes equations in two dimensions on unstructured meshes. The method is stabilized by the addition of a leastsquares operator (an inexpensive simplification of the Galerkin leastsquares method), leading to solutions free of spurious oscillations. Convergence to steady state is reached via a backward Euler timestepping scheme, and the use of local timesteps allows convergence to be accelerated. The choice of both the nonlinear solver, which is employed to solve the algebraic system arising at each timestep, and the iterative method used within this solver, is fully discussed, along with an inexpensive technique for...
A Note on Line Search Termination Criteria for Collinear Scaling Algorithms
, 2001
"... Ariyawansa [2] has introduced a line search termination criterion for collinear scaling minimization algorithms. However, he verified that this criterion provides sufficient decrease only in the case of objective functions that satisfy certain strong convexity assumptions. In this paper, we repo ..."
Abstract

Cited by 1 (1 self)
 Add to MetaCart
Ariyawansa [2] has introduced a line search termination criterion for collinear scaling minimization algorithms. However, he verified that this criterion provides sufficient decrease only in the case of objective functions that satisfy certain strong convexity assumptions. In this paper, we report results of our attempts to relax these assumptions. We present an example of a continuously differentiable, nonconvex function with bounded level sets on which the criterion of Ariyawansa [2] does not provide sufficient decrease. This example indicates that specifying line search termination criteria for collinear scaling algorithms is still an open problem. On a more positive note, we are able to show that the class of objective functions on which the criterion of Ariyawansa [2] provides sufficient decrease can be extended to include convex and strictly pseudoconvex functions.
NUMERICAL INTEGRATION METHODS
, 2002
"... Methods. (Under the direction of C. T. Kelley). Numerical methods for integrating partial differential equations are used in nearly every scientific field. In this dissertation we study two types of numerical integration methods, transient methods and pseudotransient methods. Transient methods for ..."
Abstract
 Add to MetaCart
Methods. (Under the direction of C. T. Kelley). Numerical methods for integrating partial differential equations are used in nearly every scientific field. In this dissertation we study two types of numerical integration methods, transient methods and pseudotransient methods. Transient methods for partial differential equations look for timeaccurate solutions that explain the evolution of the equation (although a steady state solution may evolve). Pseudotransient methods look for steadystate solutions of partial differential equations while paying attention to the transient behavior to aid in stability. In contrast, rootfinding methods, e.g. linesearch methods, look only for a steadystate solution often not paying attention at all to the transient behavior of the problem. Pseudotransient continuation is a method for solving steady state solutions of partial differential equations, and is used when traditional linesearch methods fail to converge or converge to nonphysical solutions. The method is a hybrid between implicit Euler and Newtonâ€™s method where variable stepsizes are used to transfer from one methodto the other. We demonstrate the performance of pseudotransient continuation both numerically and theoretically on a variety of problems. We extend the global convergence theory, which currently covers a class of ordinary differential equations, to include a class of semiexplicit index1 differentialalgebraic equations. We also studied CVode, a transient code for solving nonlinear partial differential equations. In this work, we explain how CVode was extended to allow for a twogrid nonlinear solver. The twogrid solver coarsens a given mesh and solves the nonlinear problem on the coarse mesh, which is then moved back to the fine mesh for refining. This approach can be less expensive than computing the full nonlinear solution on the fine mesh. We explore some of the theoretical and computational issues involved in implementing this method for a radiative transfer problem as might be seen in stellar fusion. TEMPORAL AND PSEUDOTEMPORAL