On-line first order backpropagation is sufficiently fast and effective for many large-scale classification problems but for very high precision mappings, batch processing may be the method of choice. This paper reviews first- and second-order optimization methods for learning in feedforward neural networks. The viewpoint is that of optimization: many methods can be cast in the language of optimization techniques, allowing the transfer to neural nets of detailed results about computational complexity and safety procedures to ensure convergence and to avoid numerical problems. The review is not intended to deliver detailed prescriptions for the most appropriate methods in specific applications, but to illustrate the main characteristics of the different methods and their mutual relations.

rence engine. This fuzzy controller then replaces the neural smith. This method- ology for choosing training parameters can be applied to other neural net- works, including Kohonen's self-organizing maps 3 and layered percepttons trained by other methods, such as random search? But beyond neural ncts, this research has led us to adopt the printpies of fizzy logic in a way that can potentially ,be broadly applied to a xvide variety of algorithms used in adap- tation and optimization. SPRING ] 996 070 9924/96,/$5.00 1996 IEEE 5 7 eFUZZ PARAMETER OPTIMIZATION Table 1. Performance measures and parameters for different neural architectures. Fuzzy parameter adaptation A fuzzy controller consists of a set of fuzzy implications of the type "IRA Then B." Consider for example the case of a single input, single-output system and suppose there are Nsuch implications. Each of these rules associates a fuzzy input subset to a fuzzy output subset, represented by their membership functions. Fuzzy set