A new method, the sequential subspace method (SSM), is developed for the problem of minimizing a quadratic over a sphere. In our scheme, the quadratic is minimized over a subspace which is adjusted in successive iterations to ensure convergence to an optimum. When a sequential quadratic programming iterate is included in the subspace, convergence is locally quadratic. Numerical comparisons with other recent methods are given.

Abstract. Total least squares (TLS) is a method for treating an overdetermined system of linear equations Ax ≈ b, where both the matrix A and the vector b are contaminated by noise. Tikhonov regularization of the TLS (TRTLS) leads to an optimization problem of minimizing the sum of fractional quadratic and quadratic functions. As such, the problem is nonconvex. We show how to reduce the problem to a single variable minimization of a function G over a closed interval. Computing a value and a derivative of G consists of solving a single trust region subproblem. For the special case of regularization with a squared Euclidean norm we show that G is unimodal and provide an alternative algorithm, which requires only one spectral decomposition. A numerical example is given to illustrate the effectiveness of our method.

Each step of an interior point method for nonlinear optimization requires the solution of a symmetric indefinite linear system known as a KKT system, or more generally, a saddle point problem. As the problem size increases, direct methods become prohibitively expensive to use for solving these problems; this leads to iterative solvers being the only viable alternative. In this thesis we consider iterative methods for solving saddle point systems and show that a projected preconditioned conjugate gradient method can be applied to these indefinite systems. Such a method requires the use of a specific class of preconditioners, (extended) constraint preconditioners, which exactly replicate some parts of the saddle point system that we wish to solve. The standard method for using constraint preconditioners, at least in the optimization community, has been to choose the constraint

Abstract. In an earlier paper [Minimizing a quadratic over a sphere, SIAM J. Optim., 12 (2001), 188–208], we presented the sequential subspace method (SSM) for minimizing a quadratic over a sphere. This method generates approximations to a minimizer by carrying out the minimization over a sequence of subspaces that are adjusted after each iterate is computed. We showed in this earlier paper that when the subspace contains a vector obtained by applying one step of Newton’s method to the first-order optimality system, SSM is locally, quadratically convergent, even when the original problem is degenerate with multiple solutions and with a singular Jacobian in the optimality system. In this paper, we prove (nonlocal) convergence of SSM to a global minimizer whenever each SSM subspace contains the following three vectors: (i) the current iterate, (ii) the gradient of the cost function evaluated at the current iterate, and (iii) an eigenvector associated with the smallest eigenvalue of the cost function Hessian. For nondegenerate problems, the convergence rate is at least linear when vectors (i)–(iii) are included in the SSM subspace. 1.

Trust region subproblems arise within a class of unconstrained methods called trust region methods. The subproblems consist of minimizing a quadratic function subject to a norm constraint. This thesis is a survey of dierent methods developed to nd an approximate solution to the subproblem. We study the well-known method of More and Sorensen [18] and two recent methods for large sparse subproblems: the so-called Lanczos method of Gould et al. [7] and the Rendl and Wolkowicz algorithm [31]. The common ground to explore these methods will be semidenite programming. This approach has been used by Rendl and Wolkowicz [31] to explain their method and the More and Sorensen algorithm; we extend this work to the Lanczos method. The last chapter of this thesis is dedicated to some improvements done to the Rendl and Wolkowicz algorithm and to comparisons between the Lanczos method and the Rendl and Wolkowicz algorithm. In particular, we show some weakness of the Lanczos method and show that ...

We present a new matrix-free method for the large-scale trust-region subproblem, assuming that the approximate Hessian is updated by the L-BFGS formula with m = 1 or 2. We determine via simple formulas the eigenvalues of these matrices and, at each iteration, we construct a positive definite matrix whose inverse can be expressed analytically, without using factorization. Consequently, a direction of negative curvature can be computed immediately by applying the inverse power method. The computation of the trial step is obtained by performing a sequence of inner products and vector summations. Furthermore, it immediately follows that the strong convergence properties of trust region methods are preserved. Numerical results are also presented.

Stability of block LDL T factorization of a

For symmetric indefinite tridiagonal matrices, block LDL T factorization without interchanges is shown to have excellent numerical stability when a pivoting strategy of Bunch is used to choose the dimension (1 or 2) of the pivots. Key words. tridiagonal matrix, symmetric indefinite matrix, diagonal pivoting method, LDL T factorization, growth factor, numerical stability, rounding error analysis, LAPACK, LINPACK. AMS subject classifications. primary 65F05, 65G05 1 Introduction Linear systems involving symmetric indefinite tridiagonal matrices arise in a number of situations. For example, Aasen's method with partial pivoting [1] produces a factorization \PiA\Pi T = LTL T of a symmetric matrix A, where \Pi is a permutation matrix, L is unit lower triangular, and T is tridiagonal. To solve a linear system Ax = b using Aasen's method it is necessary to solve a system with coefficient matrix T . A recent application that produces linear systems with symmetric tridiagonal coeffici...

Trial changes to the variables in many trust region algorithms for optimization calculations are derived from the following problem. Minimize a quadratic function Q(d), d 2 R , subject to kdk \Delta, where Q and \Delta 2 R are given, where the vector norm is Euclidean, and where n is the number of variables. Each trial change is an estimate of a solution of this problem, some estimates being better than others. We argue that the estimate of Mor'e and Sorensen (1983) should be employed more often, because it gives a value of Q(d) that differs from the optimal value by at most a prescribed tolerance. On the other hand, some researchers believe that achieving this property requires an excessive amount of computation. Therefore the details of the method are reviewed. We note that the tolerance can usually be satisfied by very few cycles of an iterative procedure. Each cycle requires the solution of a system of equations of the form (r Q+I) d = \GammarQ(0), where is a nonnegative parameter that is adjusted by each cycle. Thus the amount of computation is of magnitude n for general r Q. When a sequence of trust region problems is solved, however, and when each new r Q differs from the previous one by a matrix of rank one or two, which happens in quasi-Newton algorithms for unconstrained optimization, then the work of each iteration can be reduced to O(n ) by applying orthogonal similarity transformations to r Q (Powell, 1997). This technique is described. It should allow the Mor'e and Sorensen method to be used routinely for much larger values of n than before.

A conjugate gradient (CG)-type algorithm CG Plan is introduced for calculating an approximate solution of Newton’s equation within large-scale optimization frameworks. The approximate solution must satisfy suitable properties to ensure global convergence. In practice, the CG algorithm is widely used, but it is not suitable when the Hessian matrix is indefinite, as it can stop prematurely. CG Plan is a symmetric variant of the composite step Bi-CG method of Bank and Chan, suitably adapted for optimization problems. It is an alternative to CG that copes with the indefinite case. We show convergence for CG Plan, then prove that the practical implementation always provides a gradient related direction within a truncated Newton method (algorithm TN Plan). Some preliminary numerical results support the theory.