A new methodology of extraction, optimization, and application of sets of logical rules is described. Neural networks are used for initial rule extraction, local, or global minimization procedures for optimization, and Gaussian uncertainties of measurements are assumed during application of logical rules. Algorithms for extraction of logical rules from data with real-valued features require determination of linguistic variables or membership functions. Context-dependent membership functions for crisp and fuzzy linguistic variables are introduced and methods of their determination described. Several neural and machine learning methods of logical rule extraction generating initial rules are described, based on constrained multilayer perceptron, networks with localized transfer functions or on separability criteria for determination of linguistic variables. A tradeoff between accuracy/simplicity is explored at the rule extraction stage and between rejection/error level at the optimization stage. Gaussian uncertainties of measurements are assumed during application of crisp logical rules, leading to “soft trapezoidal” membership functions and allowing to optimize the linguistic variables using gradient procedures. Numerous applications of this methodology to benchmark and real-life problems are reported and very simple crisp logical rules for many datasets provided.

Abstract not found

The Rprop algorithm proposed by Riedmiller and Braun is one of the best performing first-order learning methods for neural networks. We introduce modifications of the algorithm that improve its learning speed. The resulting speedup is experimentally shown for a set of neural network learning tasks as well as for artificial error surfaces.

In this paper, we present MLEANN (Meta-Learning Evolutionary Artificial Neural Network), an automatic computational framework for the adaptive optimization of artificial neural networks wherein the neural network architecture, activation function, connection weights; learning algorithm and its parameters are adapted according to the problem. We explored the performance of MLEANN and conventionally designed artificial neural networks for function approximation problems. To evaluate the comparative performance, we used three different well-known chaotic time series. We also present the state of the art popular neural network learning algorithms and some experimentation results related to convergence speed and generalization performance. We explored the performance of backpropagation algorithm; conjugate gradient algorithm, quasi-Newton algorithm and Levenberg-Marquardt algorithm for the three chaotic time series. Performances of the different learning algorithms were evaluated when the activation functions and architecture were changed. We further present the theoretical background, algorithm, design strategy and further demonstrate how effective and inevitable is the proposed MLEANN framework to design a neural network, which is smaller, faster and with a better generalization performance.

... This paper is focused on the extraction and use of logical rules for data understanding. All aspects of rule generation, optimization, and application are described, including the problem of finding good symbolic descriptors for continuous data, tradeoffs between accuracy and simplicity at the rule-extraction stage, and tradeoffs between rejection and error level at the rule optimization stage. Stability of rule-based description, calculation of probabilities from rules, and other related issues are also discussed. Major approaches to extraction of logical rules based on neural networks, decision trees, machine learning, and statistical methods are introduced. Optimization and application issues for sets of logical rules are described. Applications of such methods to benchmark and real-life problems are reported and illustrated with simple logical rules for many datasets. Challenges and new directions for research are outlined.

This paper demonstrates how the backpropagation algorithm #BP# and its variants can

. $ FRPSUHKHQVLYH PHWKRGRORJ\ RI H[WUDFWLRQ RI RSWLPDO VHWV RI ORJLFDO UXOHV XVLQJ QHXUDO QHWZRUNV DQG JOREDO PLQLPL]DWLRQ SURFHGXUHV KDV EHHQ GHYHO# RSHG# ,QLWLDO UXOHV DUH H[WUDFWHG XVLQJ GHQVLW\ HVWLPDWLRQ QHXUDO QHWZRUNV ZLWK UHFWDQJXODU IXQFWLRQV RU PXOWL#OD\HUHG SHUFHSWURQ #0/3# QHWZRUNV WUDLQHG ZLWK FRQVWUDLQHG EDFNSURSDJDWLRQ DOJRULWKP# WUDQVIRUPLQJ 0/3V LQWR VLPSOHU QHWZRUNV SHUIRUPLQJ ORJLFDO IXQFWLRQV# $ FRQVWUXFWLYH DOJRULWKP FDOOHG &#0/3#/1 LV SUR# SRVHG# LQ ZKLFK UXOHV RI LQFUHDVLQJ VSHFLILFLW\ DUH JHQHUDWHG FRQVHFXWLYHO\ E\ DGGLQJ PRUH QRGHV WR WKH QHWZRUN# 1HXUDO UXOH H[WUDFWLRQ LV IROORZHG E\ RSWLPL# ]DWLRQ RI UXOHV XVLQJ JOREDO PLQLPL]DWLRQ WHFKQLTXHV# (VWLPDWLRQ RI FRQILGHQFH RI YDULRXV VHWV RI UXOHV LV GLVFXVVHG# 7KH K\EULG DSSURDFK WR UXOH H[WUDFWLRQ KDV EHHQ DSSOLHG WR D QXPEHU RI EHQFKPDUN DQG UHDO OLIH SUREOHPV ZLWK YHU\ JRRG UHVXOWV# ,Q PDQ\ FDVHV FULVS ORJLFDO UXOHV DUH TXLWH VDWLVIDFWRU\# EXW VRPHWLPHV IX]]\ UXOHV PD\ EH VLJQLILFDQWO\ PRUH DFFXUDWH# Keywords...

: The Langevin updating rule, in which noise is added to the weights during learning, is presented and analyzed. It is well controlled and, being a natural extension to standard backpropagation learning, easily combined with other modifications of backpropagation. If the Hessian matrix is numerically ill-conditioned, Langevin updating converges faster than backpropagation and, probably, also higher order algorithms. This is particularly important for multilayer perceptrons with many hidden layers, which tend to have ill-conditioned Hessians. In addition, Manhattan updating is shown to have a similar effect as Langevin updating. 1 denni@thep.lu.se Introduction Performances of artificial neural networks (ANN) are often improved when external noise is present during the training phase. For instance in Hopfield-type networks the basins of attraction for the stored memory patterns are enlarged when noise-corrupted training patterns are used [1]. In linear perceptrons the generalization a...

Selection of the topology of a network and correct parameters for the learning algorithm is a tedious task for designing an optimal Artificial Neural Network (ANN), which is smaller, faster and with a better generalization performance. Genetic algorithm (GA) is an adaptive search technique based on the principles and mechanisms of natural selection and survival of the fittest from natural evolution. Simulated annealing (SA) is a global optimization algorithm that can process cost functions possessing quite arbitrary degrees of nonlinearities, discontinuities and stochasticity but statistically assuring a optimal solution. In this paper we explain how a hybrid algorithm integrating the desirable aspects of GA and SA can be applied for the optimal design of an ANN. This paper is more concerned with the understanding of current theoretical developments of Evolutionary Artificial Neural Networks (EANNs) using GAs and other heuristic procedures and how the proposed hybrid and other heuristic procedures can be combined to produce an optimal ANN.

Methodology of extraction of optimal sets of logical rules using neural networks and global minimization procedures has been developed. Initial rules are extracted using density estimation neural networks with rectangular functions or multi-layered perceptron (MLP) networks trained with constrained backpropagation algorithm, transforming MLPs into simpler networks performing logical functions. A constructive algorithm called C-MLP2LN is proposed, in which rules of increasing specificity are generated consecutively by adding more nodes to the network. Neural rule extraction is followed by optimization of rules using global minimization techniques. Estimation of confidence of various sets of rules is discussed. The hybrid approach to rule extraction has been applied to a number of benchmark and real life problems with very good results. Keywords: computational intelligence, neural networks, extraction of logical rules, data mining 1 Logical rules - introduction. Why should one use log...