On-line first order backpropagation is sufficiently fast and effective for many large-scale classification problems but for very high precision mappings, batch processing may be the method of choice. This paper reviews first- and second-order optimization methods for learning in feedforward neural networks. The viewpoint is that of optimization: many methods can be cast in the language of optimization techniques, allowing the transfer to neural nets of detailed results about computational complexity and safety procedures to ensure convergence and to avoid numerical problems. The review is not intended to deliver detailed prescriptions for the most appropriate methods in specific applications, but to illustrate the main characteristics of the different methods and their mutual relations.

. This paper describes an approach to constructing derivative-free algorithms for unconstrained optimization that are easy to implement on parallel machines. A special feature of this approach is the ease with which algorithms can be generated to take advantage of any number of processors and to adapt to any cost ratio of communication to function evaluation. Numerical tests show speed-ups on two fronts. The cost of synchronization being minimal, the speed-up is almost linear with the addition of more processors, i.e., given a problem and a search strategy, the decrease in execution time is proportional to the number of processors added. Even more encouraging, however, is that different search strategies, devised to take advantage of additional (or more powerful) processors, may actually lead to dramatic improvements in the performance of the basic algorithm. Thus search strategies intended for many processors actually may generate algorithms that are better even when implemented seque...

This report tells how to make solvers work with AMPL's solve command. It describes an interface library, amplsolver.a, whose source is available from netlib. Examples include programs for listing LPs, automatic conversion to the LP dual (shellscript as solver), solvers for various nonlinear problems (with first and sometimes second derivatives computed by automatic differentiation), and getting C or Fortran 77 for nonlinear constraints, objectives and their first derivatives. Drivers for various well known linear, mixed-integer, and nonlinear solvers provide more examples.

this paper, we consider two approaches to computing the desired implicitly defined derivative x

We describe favorable experience with automatic differentiation of mathematical programming problems expressed in AMPL, a modeling language for mathematical programming. Nonlinear expressions are translated to loop-free code, which makes analytically correct gradients and Jacobians particularly easy to compute -- static storage allocation suffices. The nonlinear expressions may either be interpreted or, to gain some execution speed, converted to Fortran or C.

Despite the widespread popularity of online opinion forums among consumers, the business value that such systems bring to organizations has, so far, remained an unanswered question. This paper addresses this question by studying the value of online movie ratings in forecasting motion picture revenues. First, we conduct a survey where a nationally representative sample of subjects who do not rate movies online is asked to rate a number of recent movies. Their ratings exhibit high correlation with online ratings for the same movies. We thus provide evidence for the claim that online ratings can be considered as a useful proxy for word-of-mouth about movies. Inspired by the Bass model of product diffusion, we then develop a motion picture revenue-forecasting model that incorporates the impact of both publicity and word of mouth on a movie's revenue trajectory. Using our model, we derive notably accurate predictions of a movie's total revenues from statistics of user reviews posted on Yahoo! Movies during the first week of a new movie's release. The results of our work provide encouraging evidence for the value of publicly available online forum information to firms for real-time forecasting and competitive analysis.

this paper to identify both approaches. The codes possess the following additional features: 1. Scaling. For all practical parameter-estimation problems, scaling of model function values is very important because of the variation of the experimental numerical data. Thus SYSFIT proceeds from scaling parameters

When nonlinear least squares problems are separable, i.e., some parameters appear linearly, it is often useful to project the linear parameters out of the problem, which leaves a nonlinear least-squares problem involving only the nonlinear parameters. For large-residual problems, it saves iterations but costs work and storage to use an "augmented model" in solving this problem. When there are multiple righthand sides, the costs can be large, making a Levenberg-Marquardt or even a normal-equations algorithm more attractive; for some such problems, the associative law and extra orthogonal transformations can significantly reduce operation counts. Nonlinear least-squares problems often have the form minimize f(c , x) : = 1 /2 ïïF(x) c - y(x) ïï 2 2 , where F: IR p ® IR n× and y: IR p ® IR n are continuously differentiable. As Golub and Pereyra [5, 6] have shown, minimizing f(c , x) is equivalent to minimizing f (x) : = 1 /2 ïïP F ^ y(x) ïï 2 2 , (1) where P F ^ = P F(x...

This paper introduces a new method of registering point sets. The registration error is directly minimized using general-purpose nonlinear optimization (the Levenberg-Marquardt algorithm). The surprising conclusion of the paper is that this technique is comparable in speed to the special-purpose ICP algorithm which is most commonly used for this task. Because the routine directly minimizes an energy function, it is easy to extend it to incorporate robust estimation via a Huber kernel, yielding a basin of convergence that is many times wider than existing techniques. Finally we introduce a data structure for the minimization based on the chamfer distance transform which yields an algorithm which is both faster and more robust than previously described methods. 1

ii ii The well known Levenberg-Marquardt method is used extensively for solving nonlinear least-squares problems. We describe an extension of the Levenberg-Marquardt method to problems with bound constraints on the variables. Each iteration of our algorithm approximately solves a linear least-squares problem subject to the original bound constraints. Our approach is especially suited to large-scale problems whose functions are expensive to compute; only matrixvector products with the Jacobian are required. We present the results of numerical experiments that illustrate the effectiveness of the approach. Moreover, we describe its application to a practical curve fitting problem in fluorescence optical imaging. iii Contents