Abstract. Direct search methods are best known as unconstrained optimization techniques that do not explicitly use derivatives. Direct search methods were formally proposed and widely applied in the 1960s but fell out of favor with the mathematical optimization community by the early 1970s because they lacked coherent mathematical analysis. Nonetheless, users remained loyal to these methods, most of which were easy to program, some of which were reliable. In the past fifteen years, these methods have seen a revival due, in part, to the appearance of mathematical analysis, as well as to interest in parallel and distributed computing. This review begins by briefly summarizing the history of direct search methods and considering the special properties of problems for which they are well suited. Our focus then turns to a broad class of methods for which we provide a unifying framework that lends itself to a variety of convergence results. The underlying principles allow generalization to handle bound constraints and linear constraints. We also discuss extensions to problems with nonlinear constraints.

We address the question: How should N n-dimensional subspaces of m-dimensional Euclidean space be arranged so that they are as far apart as possible? The results of extensive computations for modest values of N; n; m are described, as well as a reformulation of the problem that was suggested by these computations. The reformulation gives a way to describe n- dimensional subspaces of m-space as points on a sphere in dimension (m \Gamma 1)(m+2), which provides a (usually) lowerdimensional representation than the Pl ucker embedding, and leads to a proof that many of the new packings are optimal. The results have applications to the graphical display of multidimensional data via Asimov's grand tour method.

We propose a new method for the intermodal registration of images using a criterion known as mutual information. Our main contribution is an optimizer that we specifically designed for this criterion. We show that this new optimizer is well adapted to a multiresolution approach because it typically converges in fewer criterion evaluations than other optimizers. We have built a multiresolution image pyramid, along with an interpolation process, an optimizer, and the criterion itself, around the unifying concept of spline-processing. This ensures coherence in the way we model data and yields good performance. We have tested our approach in a variety of experimental conditions and report excellent results. We claim an accuracy of about a hundredth of a pixel under ideal conditions. We are also robust since the accuracy is still about a tenth of a pixel under very noisy conditions. In addition, a blind evaluation of our results compares very favorably to the work of several other researchers.

The need to optimize a function whose derivatives are unknown or non-existent arises in many contexts, particularly in real-world applications. Various direct search methods, most notably the Nelder-Mead `simplex' method, were proposed in the early 1960s for such problems, and have been enormously popular with practitioners ever since. Nonetheless, for more than twenty years these methods were typically dismissed or ignored in the mainstream optimization literature, primarily because of the lack of rigorous convergence results. Since 1989, however, direct search methods have been rejuvenated and made respectable. This paper summarizes the history of direct search methods, with special emphasis on the Nelder-Mead method, and describes recent work in this area. This paper is based on a plenary talk given at the Biennial Dundee Conference on Numerical Analysis, Dundee, Scotland, 1995. 1. Introduction Unconstrained optimization---the problem of minimizing a nonlinear function f(x) for x 2...

Motivation: Simulation and modeling is becoming a standard approach to understand complex biochemical processes. Therefore, there is a big need for software tools that allow access to diverse simulation and modeling methods as well as support for the usage of these methods. Results: Here, we present COPASI, a platform-independent and user-friendly biochemical simulator that offers several unique features. We discuss numerical issues with these features, in particular the criteria to switch between stochastic and deterministic simulation methods, hybrid deterministic-stochastic methods, and the importance of random number generator numerical resolution in stochastic simulation. Availability: The complete software is available in binary (executable) for MS Windows, OS X, Linux (Intel), and Sun Solaris (SPARC), as well as the full source code under an open source license from

We discuss direct search methods for unconstrained optimization. We give a modern perspective on this classical family of derivative-free algorithms, focusing on the development of direct search methods during their golden age from 1960 to 1971. We discuss how direct search methods are characterized by the absence of the construction of a model of the objective. We then consider a number of the classical direct search methods and discuss what research in the intervening years has uncovered about these algorithms. In particular, while the original direct search methods were consciously based on straightforward heuristics, more recent analysis has shown that in most — but not all — cases these heuristics actually

: Many different procedures have been proposed for optimization calculations when first derivatives are not available. Further, several researchers have contributed to the subject, including some who wish to prove convergence theorems, and some who wish to make any reduction in the least calculated value of the objective function. There is not even a key idea that can be used as a foundation of a review, except for the problem itself, which is the adjustment of variables so that a function becomes least, where each value of the function is returned by a subroutine for each trial vector of variables. Therefore the paper is a collection of essays on particular strategies and algorithms, in order to consider the advantages, limitations and theory of several techniques. The subjects that are addressed are line search methods, the restriction of vectors of variables to discrete grids, the use of geometric simplices, conjugate direction procedures, trust region algorithms that form linear or...

This paper presents a method for planning the motions of autonomous vehicles moving on general terrains. The method obtains the geometric path and vehicle speeds that minimize motion time consid- ering vehicle dynamics, terrain topography, obstacles, and surface mobility. The terrain is represented by a smooth cubic B patch, and the geometric path consists of a B spline curve mapped to the surface. The time optimal motions are computed by first obtaining the best ohstacle- free path from all paths represented by a uniform grid. This path is further optimized with a local optimization, using the optimal motion time along the path as the cost function and the control points of a B spline as the optimizing parameters. Examples are presented that demonstrate the method for a simple dynamic model of a vehicle moving on a mountainous terrain. I. INTRODUCTION T HE problem of motion planning of autonomous vehicles consistsof selecting the geometric path and vehicle speeds so as to avoi...

We present two new classes of pattern search algorithms for unconstrained min-imization: the rank ordered and the positive basis pattern search methods. These algorithms can nearly halve the worst case cost of an iteration compared to the classi-cal pattern search algorithms. The rank ordered pattern search methods are based on a heuristic for approximating the direction of steepest descent, while the positive basis pattern search methods are motivated by a generalization of the geometry characteris-tic of the patterns of the classical methods. We describe the new classes of algorithms and present the attendant global convergence analysis. * This research was supported by the National Aeronautics and Space Administration under NASA

Evolution strategies are general, nature-inspired heuristics for search and optimization. Supported both by empirical evidence and by recent theoretical findings, there is a common belief that evolution strategies are robust and reliable, and frequently they are the method of choice if neither derivatives of the objective function are at hand nor differentiability and numerical accuracy can be assumed. However, despite their widespread use, there is little exchange between members of the “classical ” optimization community and people working in the field of evolutionary computation. It is our belief that both sides would benefit from such an exchange. In this paper, we present a brief outline of evolution strategies and discuss some of their properties in the presence of noise. We then empirically demonstrate that for a simple but nonetheless nontrivial noisy objective function, an evolution strategy outperforms other optimization algorithms designed to be able to cope with noise. The environment in which the algorithms are tested is deliberately chosen to afford a transparency of the results that reveals the strengths and shortcomings of the strategies, making it possible to draw conclusions with regard to the design of better optimization algorithms for noisy environments. 1