In this paper, we present MLEANN (Meta-Learning Evolutionary Artificial Neural Network), an automatic computational framework for the adaptive optimization of artificial neural networks wherein the neural network architecture, activation function, connection weights; learning algorithm and its parameters are adapted according to the problem. We explored the performance of MLEANN and conventionally designed artificial neural networks for function approximation problems. To evaluate the comparative performance, we used three different well-known chaotic time series. We also present the state of the art popular neural network learning algorithms and some experimentation results related to convergence speed and generalization performance. We explored the performance of backpropagation algorithm; conjugate gradient algorithm, quasi-Newton algorithm and Levenberg-Marquardt algorithm for the three chaotic time series. Performances of the different learning algorithms were evaluated when the activation functions and architecture were changed. We further present the theoretical background, algorithm, design strategy and further demonstrate how effective and inevitable is the proposed MLEANN framework to design a neural network, which is smaller, faster and with a better generalization performance.

Abstract—An application of numerical gradient (NG) to training of MLP networks is presented. Several versions of the algorithm and the influence of various parameters on the training process are discussed. Optimization of network parameters based on global search with numerical gradient is presented. Examples of two-dimensional projection of the error surface are shown and the influence of various numerical gradient parameters on the error surface is presented. The speed and accuracy of this method is compared with the search-based MLP training algorithm.

Backpropagation based on minimization algorithms is replaced by heuristic search techniques for quantized weights. The resulting algorithm is fast, avoids local minima of the cost function, and may be used either as initialization method for standard backpropagation or as a logical rule extraction technique.

Algorithms based on systematic search techniques can be successfully applied for multilayer perceptron (MLP) training and for logical rule extraction from data using MLP networks. The proposed solutions are easier to implement and frequently outperform gradient-based optimization algorithms. Search-based techniques, popular in artificial intelligence and almost completely neglected in neural networks can be the basis for MLP network training algorithms. There are plenty of well-known search algorithms, however since they are not suitable for MLP training, new algorithms dedicated to this task must be developed. Search algorithms applied to MLP networks change network parameters (weights and biases) and check the influence of the changes on the error function. MLP networks considered in this thesis are used for data classification and logical rule-based understanding of the data. The proposed solutions in many cases outperform gradient-based backpropagation algorithms. The thesis is organized in three parts. The first part of the thesis concentrates on better understanding of MLP properties.

Abstract. The MLP training process is analyzed and a variable step search-based algorithm (VSS) that does not require gradient information is introduced. This algorithm finds rough position of the minima in each single weight direction, and successively updates the weights. Only a small fragment of the network is analyzed for each update, making the method computationally efficient. The VSS algorithm is simpler to program than backpropagation, yet the quality of results and the speed of convergence are at the level of state-of-the-art Levenberg-Marquardt and scaled conjugate gradient algorithms. 1 Introduction. Multilayer perceptrons (MLP) are usually trained using analytical gradient-based algorithms with error backpropagation. Some of the most popular methods that include the standard backpropagation (BP), RPROP, Quickprop, Levenberg-Marquardt (LM) [1] [2], and the scaled conjugate gradient (SCG) algorithm [3] [4]. Also many global optimization

Selection of the topology of a neural network and correct parameters for the learning algorithm is a tedious task for designing an optimal artificial neural network, which is smaller, faster and with a better generalization performance. In this paper we introduce a recently developed cutting angle method (a deterministic technique) for global optimization of connection weights. Neural networks are initially trained using the cutting angle method and later the learning is fine-tuned (meta-learning) using conventional gradient descent or other optimization techniques. Experiments were carried out on three time series benchmarks and a comparison was done using evolutionary neural networks. Our preliminary experimentation results show that the proposed deterministic approach could provide near optimal results much faster than the evolutionary approach. 1.

Abstract: A new class of search-based training algorithms for feedforward networks is introduced. These algorithms do not calculate analytical gradients and do not use stochastic or genetic search techniques. The forward step is performed to calculate error in response to localized weight changes using systematic search techniques. One of the simplest variants of this type of algorithms, the Variable Step Search (VSS) algorithm, is studied in details. The VSS search procedure changes one network parameter at a time and thus does not impose any restrictions on the network structure or the type of transfer functions. Rough approximation to the gradient direction and the determination of the optimal step along this direction to find the minimum of error are performed simultaneously. Modifying the value of a single weight changes the signals only in a small fragment of the network, allowing for efficient calculations of contributions to errors. Several heuristics are discussed to increase the efficiency of VSS algorithm. Tests on benchmark data show that VSS can outperform such renown algorithms as the Levenberg-Marquardt or scaled conjugate gradient algorithm. * Manuscript