## Convergence of the restricted Nelder-Mead algorithm in two dimensions, in preparation (1997)

### Cached

### Download Links

Citations: | 1 - 1 self |

### BibTeX

@MISC{Lagarias97convergenceof,

author = {Jeffrey C. Lagarias and Bjorn Poonen and Margaret H. Wright},

title = {Convergence of the restricted Nelder-Mead algorithm in two dimensions, in preparation},

year = {1997}

}

### OpenURL

### Abstract

The Nelder–Mead algorithm, a longstanding direct search method for unconstrained optimization published in 1965, is designed to minimize a scalar-valued function f of n real variables using only function values, without any derivative information. Each Nelder–Mead iteration is associated with a nondegenerate simplex defined by n + 1 vertices and their function values; a typical iteration produces a new simplex by replacing the worst vertex by a new point. Despite the method’s widespread use, theoretical results have been limited: for strictly convex objective functions of one variable with bounded level sets, the algorithm always converges to the minimizer; for such functions of two variables, the diameter of the simplex converges to zero, but examples constructed by McKinnon show that the algorithm may converge to a nonminimizing point. This paper considers the restricted Nelder–Mead algorithm, a variant that does not allow expansion steps. In two dimensions we show that, for any nondegenerate starting simplex and any twice-continuously differentiable function with positive definite Hessian and bounded level sets, the algorithm always converges to the minimizer. The proof is based on treating the method as a discrete dynamical system, and relies on several techniques that are non-standard in convergence proofs for unconstrained optimization. 1

### Citations

1459 | Theory of linear and integer programming - Schrijver - 1986 |

1371 |
A simplex method for function minimization
- Nelder, Mead
- 1965
(Show Context)
Citation Context ... search” methods. See [5] for a recent survey of derivative-free methods; discussions focusing on direct search methods include, for example, [31, 12, 16, 14, 22]. The Nelder–Mead (NM) simplex method =-=[20]-=- is a direct search method. Each iteration of the NM method begins with a nondegenerate simplex (a geometric figure in n dimensions of nonzero volume that is the convex hull of n + 1 vertices), define... |

250 | Convergence Properties of the Nelder-Mead Simplex Method in Low Dimensions
- Lagarias, Reeds, et al.
- 1998
(Show Context)
Citation Context ...ges (see [23, 17, 7]) and its apparent usefulness in some situations. In an effort to develop positive theory about the original NM algorithm, an analysis of its convergence behavior was initiated in =-=[15]-=- in 1998, along with resolution of ambiguities in [20] about whether function comparisons involve “greater than” or “greater than or equal” tests. 1 In what follows we use the term Nelder-Mead algorit... |

164 |
Iterative Methods for Optimization
- Kelley
- 1999
(Show Context)
Citation Context ... explicitly involve such a model tend to be called “direct search” methods. See [5] for a recent survey of derivative-free methods; discussions focusing on direct search methods include, for example, =-=[31, 12, 16, 14, 22]-=-. The Nelder–Mead (NM) simplex method [20] is a direct search method. Each iteration of the NM method begins with a nondegenerate simplex (a geometric figure in n dimensions of nonzero volume that is ... |

148 | On the convergence of pattern search algorithms - Torczon - 1997 |

124 | Optimization by direct search: New perspectives on some classical and modern methods
- KOLDA, LEWIS, et al.
(Show Context)
Citation Context ... explicitly involve such a model tend to be called “direct search” methods. See [5] for a recent survey of derivative-free methods; discussions focusing on direct search methods include, for example, =-=[31, 12, 16, 14, 22]-=-. The Nelder–Mead (NM) simplex method [20] is a direct search method. Each iteration of the NM method begins with a nondegenerate simplex (a geometric figure in n dimensions of nonzero volume that is ... |

82 | Direct search methods: once scorned, now respectable
- Wright
- 1995
(Show Context)
Citation Context ... explicitly involve such a model tend to be called “direct search” methods. See [5] for a recent survey of derivative-free methods; discussions focusing on direct search methods include, for example, =-=[31, 12, 16, 14, 22]-=-. The Nelder–Mead (NM) simplex method [20] is a direct search method. Each iteration of the NM method begins with a nondegenerate simplex (a geometric figure in n dimensions of nonzero volume that is ... |

81 | E.: Mesh adaptive direct search algorithms for constrained optimization - Audet, Jr, et al. - 2006 |

61 | Direct search algorithms for optimization calculations
- Powell
- 1998
(Show Context)
Citation Context |

60 | Analysis of generalized pattern searches
- Audet, Dennis
- 2003
(Show Context)
Citation Context ...w-dimensional counterexamples (see (1.1)) have made the NM method an outlier among modern direct search methods, which are deliberately based on a rigorous mathematical foundation. (See, for example, =-=[6, 2, 14, 1]-=-, as well as more recent publications about direct search methods for constrained problems.) Nevertheless the NM method retains importance because of its continued use and availability in computer pac... |

59 |
Introduction to derivative-free optimization
- Conn, Scheinberg, et al.
- 2009
(Show Context)
Citation Context ...local model of relevant nonlinear functions are often described as “modelbased”, and derivative-free methods that do not explicitly involve such a model tend to be called “direct search” methods. See =-=[5]-=- for a recent survey of derivative-free methods; discussions focusing on direct search methods include, for example, [31, 12, 16, 14, 22]. The Nelder–Mead (NM) simplex method [20] is a direct search m... |

54 | Convergence of the Nelder-Mead simplex method to a nonstationary point
- McKinnon
- 1999
(Show Context)
Citation Context ...the simplices converges to zero. But it was not shown that the simplices always converge to a limiting point, and up to now this question remains unresolved. Taking the opposite perspective, McKinnon =-=[18]-=- devised a family of two-dimensional counterexamples consisting of strictly convex functions with bounded level sets and a specified initial simplex, for which the NM simplices converge to a nonminimi... |

49 | Elementary Mathematics from an Advanced Standpoint: Geometry - Klein - 1940 |

31 | Detection and remediation of stagnation in the Nelder-Mead algorithm using a sufficient decrease condition
- Kelley
- 1999
(Show Context)
Citation Context ...ease the size of the simplex; imposing a “sufficient decrease” condition (stronger than simple decrease) for accepting a new vertex; and resetting the simplex to one that is “nice”. See, for example, =-=[25, 30, 29, 11, 24, 19, 5]-=-, a small selection of the many papers that include convergence results for modifications of Nelder–Mead. Our object in this paper is to fill in additional theory for the NM algorithm in the two-dimen... |

26 | On the convergence of grid-based methods for unconstrained optimization
- COOPE, PRICE
- 2001
(Show Context)
Citation Context ...w-dimensional counterexamples (see (1.1)) have made the NM method an outlier among modern direct search methods, which are deliberately based on a rigorous mathematical foundation. (See, for example, =-=[6, 2, 14, 1]-=-, as well as more recent publications about direct search methods for constrained problems.) Nevertheless the NM method retains importance because of its continued use and availability in computer pac... |

21 | Convergence results for pattern search algorithms are tight
- Audet
(Show Context)
Citation Context ...w-dimensional counterexamples (see (1.1)) have made the NM method an outlier among modern direct search methods, which are deliberately based on a rigorous mathematical foundation. (See, for example, =-=[6, 2, 14, 1]-=-, as well as more recent publications about direct search methods for constrained problems.) Nevertheless the NM method retains importance because of its continued use and availability in computer pac... |

15 | Fortified-descent simplicial search method: a general approach
- Tseng
- 1999
(Show Context)
Citation Context ...ease the size of the simplex; imposing a “sufficient decrease” condition (stronger than simple decrease) for accepting a new vertex; and resetting the simplex to one that is “nice”. See, for example, =-=[25, 30, 29, 11, 24, 19, 5]-=-, a small selection of the many papers that include convergence results for modifications of Nelder–Mead. Our object in this paper is to fill in additional theory for the NM algorithm in the two-dimen... |

13 |
An interactive approach for solving multi-objective optimization problems. Unpublished doctoral dissertation, Univesity Microfilms international
- Woods
- 1985
(Show Context)
Citation Context ...variants) for more than 45 years, in many scientific and engineering applications. But little mathematical analysis of any kind of the method’s performance has appeared, with a few exceptions such as =-=[30, 10]-=- (from more than 20 years ago) and (more recently) [9]. As we discuss in more detail below, obtaining even limited convergence proofs for the original method has turned out to be far from simple. The ... |

11 |
A convergent variant of the Nelder-Mead algorithm
- Price, Coope, et al.
(Show Context)
Citation Context ...ease the size of the simplex; imposing a “sufficient decrease” condition (stronger than simple decrease) for accepting a new vertex; and resetting the simplex to one that is “nice”. See, for example, =-=[25, 30, 29, 11, 24, 19, 5]-=-, a small selection of the many papers that include convergence results for modifications of Nelder–Mead. Our object in this paper is to fill in additional theory for the NM algorithm in the two-dimen... |

7 |
A Teukolsky, W T Vettering and
- Press, S
- 1992
(Show Context)
Citation Context ...s more recent publications about direct search methods for constrained problems.) Nevertheless the NM method retains importance because of its continued use and availability in computer packages (see =-=[23, 17, 7]-=-) and its apparent usefulness in some situations. In an effort to develop positive theory about the original NM algorithm, an analysis of its convergence behavior was initiated in [15] in 1998, along ... |

6 |
Simplex algorithms for unconstrained optimization
- Rykov
- 1983
(Show Context)
Citation Context |

5 | Gilding the lily: A variant of the Nelder–Mead algorithm based on golden section search
- Nazareth, Tseng
(Show Context)
Citation Context |

4 |
Effect of dimensionality on the Nelder–Mead simplex method
- Han, Neumann
- 2006
(Show Context)
Citation Context ...gineering applications. But little mathematical analysis of any kind of the method’s performance has appeared, with a few exceptions such as [30, 10] (from more than 20 years ago) and (more recently) =-=[9]-=-. As we discuss in more detail below, obtaining even limited convergence proofs for the original method has turned out to be far from simple. The shortage of theory, plus the discovery of low-dimensio... |

1 |
Gurson (2000). “Simplex search behavior in nonlinear optimization”, Bachelor’s honors thesis
- P
(Show Context)
Citation Context ...own whether the NM algorithm converges for the prototypically nice function f(x, y) = x2 + y2 . 1 Resolution of these ambiguities can have a noticeable effect on the performance of the algorithm; see =-=[8]-=-.3 1.5 1 0.5 0 −0.5 −1 −1 −0.5 0 0.5 1 1.5 Figure 1: The NM algorithm’s failure on the McKinnon counterexample (1.1). Here we answer this question affirmatively for a simplified variant of the NM alg... |

1 |
Simplex distortions in Nelder–Mead reflections, IMSL
- Hensley, Smith, et al.
- 1988
(Show Context)
Citation Context ...variants) for more than 45 years, in many scientific and engineering applications. But little mathematical analysis of any kind of the method’s performance has appeared, with a few exceptions such as =-=[30, 10]-=- (from more than 20 years ago) and (more recently) [9]. As we discuss in more detail below, obtaining even limited convergence proofs for the original method has turned out to be far from simple. The ... |

1 | Trosset (2001). Direct search methods: then and now - Lewis, Torczon, et al. |