Sequential quadratic programming (SQP) methods have proved highly effective for solving constrained optimization problems with smooth nonlinear functions in the objective and constraints. Here we consider problems with general inequality constraints (linear and nonlinear). We assume that first derivatives are available, and that the constraint gradients are sparse.

this paper we examine the underlying ideas of the SQP method and the theory that establishes it as a framework from which effective algorithms can

The problem of discriminating between two finite point sets in n-dimensional feature space by a separating plane that utilizes as few of the features as possible, is formulated as a mathematical program with a parametric objective function and linear constraints. The step function that appears in the objective function can be approximated by a sigmoid or by a concave exponential on the nonnegative real line, or it can be treated exactly by considering the equivalent linear program with equilibrium constraints (LPEC). Computational tests of these three approaches on publicly available real-world databases have been carried out and compared with an adaptation of the optimal brain damage (OBD) method for reducing neural network complexity. One feature selection algorithm via concave minimization (FSV) reduced cross-validation error on a cancer prognosis database by 35.4% while reducing problem features from 32 to 4. Feature selection is an important problem in machine learning [18, 15, 1...

. In this paper some Newton and quasi-Newton algorithms for the solution of inequality constrained minimization problems are considered. All the algorithms described produce sequences fx k g converging q-superlinearly to the solution. Furthermore, under mild assumptions, a q-quadratic convergence rate in x is also attained. Other features of these algorithms are that the solution of linear systems of equations only is required at each iteration and that the strict complementarity assumption is never invoked. First the superlinear or quadratic convergence rate of a Newton-like algorithm is proved. Then, a simpler version of this algorithm is studied and it is shown that it is superlinearly convergent. Finally, quasi-Newton versions of the previous algorithms are considered and, provided the sequence defined by the algorithms converges, a characterization of superlinear convergence extending the result of Boggs, Tolle and Wang is given. Key Words. Inequality constrained optimization, New...

The need for improvement in process operations, logistics and supply chain management has created a great demand for the development of optimization models for planning and scheduling. In this paper we first review the major classes of planning and scheduling models that arise in process operations, and establish the underlying mathematical structure of these problems. As will be shown, the nature of these models is greatly affected by the time representation (discrete or continuous), and is often dominated by discrete decisions. We then briefly review the major recent developments in mixed-integer linear and nonlinear programming, disjunctive programming and constraint programming, as well as general decomposition techniques for solving these problems. We present a general formulation for integrating planning and scheduling to illustrate the models and methods discussed in this paper.

Machine learning problems of supervised classification, unsupervised clustering and parsimonious approximation are formulated as mathematical programs. The feature selection problem arising in the supervised classification task is effectively addressed by calculating a separating plane by minimizing separation error and the number of problem features utilized. The support vector machine approach is formulated using various norms to measure the margin of separation. The clustering problem of assigning m points in n-dimensional real space to k clusters is formulated as minimizing a piecewise-linear concave function over a polyhedral set. This problem is also formulated in a novel fashion by minimizing the sum of squared distances of data points to nearest cluster planes characterizing the k clusters. The problem of obtaining a parsimonious solution to a linear system where the right hand side vector may be corrupted by noise is formulated as minimizing the system residual plus either the number of nonzero elements in the solution vector or the norm of the solution vector. The feature selection problem, the clustering problem and the parsimonious approximation problem can all be stated as the minimization of a concave function over a polyhedral region and are solved by a theoretically justifiable, fast and finite successive linearization algorithm. Numerical tests indicate the utility and efficiency of these formulations on real-world databases. In particular, the feature selection approach via concave minimization computes a separating-plane based classifier that improves upon the generalization ability of a separating plane computed without feature suppression. This approach produces ii classifiers utilizing fewer original problem features than the support vector machin...

Abstract. This paper describes a reduced quasi-Newton method for solving equality constrained optimization problems. A major difficulty encountered by this type of algorithm is the design of a consistent technique for maintaining the positive definiteness of the matrices approximating the reduced Hessian of the Lagrangian. A new approach is proposed in this paper. The idea is to search for the next iterate along a piecewise linear path. The path is designed so that some generalized Wolfe conditions can be satisfied. These conditions allow the algorithm to sustain the positive definiteness of the matrices from iteration to iteration by a mechanism that has turned out to be efficient in unconstrained optimization.

Abstract. A technique for maintaining the positive definiteness of the matrices in the quasi-Newton version of the SQP algorithm is proposed. In our algorithm, matrices approximating the Hessian of the augmented Lagrangian are updated. The positive definiteness of these matrices in the space tangent to the constraint manifold is ensured by a so-called piecewise line-search technique, while their positive definiteness in a complementary subspace is obtained by setting the augmentation parameter. In our experiment, the combination of these two ideas leads to a new algorithm that turns out to be more robust and often improves the results obtained with other approaches.

We consider an algorithm for solving the optimization problem under convex constraints. Although the convexity of the constraints is treated in its generality, in practice, convex constraints imply linear inequalities, especially upper and lower bounds on the decision variables. Apart from convexity, the main assumption on the constraints is that they are once differentiable. The method belongs to the powerful class of algorithms developed by Goldstein-Levitin-Polyak. It requires that the constraints are satisfied at every iteration. This is an inconvenience in general. Yet, for linear constraints and for special constraint structures, it is simple to maintain feasibility. Two important advantage that seems to be offered in return for satisfying the constraints are the strong results concerning unit stepsize achievement and superlinear convergence. With regard to the latter, it is established that the necessary and sufficient condition for Q-superlinear rate of convergence is the two sided projected Hessian condition which, in other algorithms, can only ensure lesser Q-superlinear rates.

Penalty and barrier methods are indirect ways of solving constrained optimization problems, using techniques developed in the unconstrained optimization realm. In what follows we shall give the foundation of two more direct ways of solving constrained optimization problems, namely Sequential Quadratic Programming (SQP) methods and Primal-Dual Interior Point (PDIP) methods. 1 Sequential Quadratic Programming For the derivation of the Sequential Quadratic Programming method we shall use the equality constrained problem minimize f(x) x subject to h(x) = 0, (ECP) where f: R n → R and h: R n → R m are smooth functions. An understanding of this problem is essential in the design of SQP methods for general nonlinear programming problems. The KKT conditions for this problem are given by ∇f(x) + m� λi∇hi(x) = 0 (1a) i=1 1 h(x) = 0 (1b) where λ ∈ R m are the Lagrange multipliers associated with the equality constraints. If we use the Lagrangian L(x, λ) = f(x) + m� λihi(x) (2) we can write the KKT conditions (1) more compactly as ∇x L(x, λ) = 0. (EQKKT) ∇λ L(x, λ) As with Newton’s method unconstrained optimization, the main idead behind SQP is to model problem (ECP) at a given point x (k) by a quadratic programming subrpoblem and then use the solution to this problem to construct a more accurate approximation x (k+1). If we perform a Taylor series expansion of system (EQKKT) about (x (k) , λ (k) ) we obtain ∇x L(x (k) , λ (k))