Constructing models from time series with nontrivial dynamics involves the problem of how to choose the best model from within a class of models, or to choose between competing classes. This paper discusses a method of building nonlinear models of possibly chaotic systems from data, while maintaining good robustness against noise. The models that are built are close to the simplest possible according to a description length criterion. The method will deliver a linear model if that has shorter description length than a nonlinear model. We show how our models can be used for prediction, smoothing and interpolation in the usual way. We also show how to apply the results to identification of chaos by detecting the presence of homoclinic orbits directly from time series. 1 The Model Selection Problem As our understanding of chaotic and other nonlinear phenomena has grown, it has become apparent that linear models are inadequate to model most dynamical processes. Nevertheless, linear models...

We propose a new learning algorithm for regression modeling. The method is especially suitable for optimizing neural network structures that are amenable to a statistical description as mixture models. These include mixture of experts, hierarchical mixture of experts (HME), and normalized radial basis functions (NRBF). Unlike recent maximum likelihood (ML) approaches, we directly minimize the (squared) regression error. We use the probabilistic framework as means to define an optimization method that avoids many shallow local minima on the complex cost surface. Our method is based on deterministic annealing (DA), where the entropy of the system is gradually reduced, with the expected regression cost (energy) minimized at each entropy level. The corresponding Lagrangian is the system's "free-energy," and this annealing process is controlled by variation of the Lagrange multiplier, which acts as a "temperature" parameter. The new method consistently and substantially outperformed the com...

Abstract-Artificial neural networks (ANNs) have been used to construct empirical nonlinear models of process data. Because network models are not based on physical theory and contain nonlinearities, their predictions are suspect when extrapolating beyond the range of the original training data. With multiple correlated inputs, it is difficult to recognize when the network is extrapolating. Furthermore, due to nonuniform distribution of the training examples and noise over the domain, the network may have local areas of poor fit even when not extrapolating. Standard measures of network performance give no indication of regions of locally poor fit or possible errors due to extrapolation. This paper introduces the "validity index network " (VI-net), an extension of radial basis function networks (RBFN), that calculates the reliability and the confidence of its output and indicates local regions of poor fit and extrapolation. Because RBFNs use a composition of local fits to the data, they are readily adapted to predict local fitting accuracy. The VI-net can also detect novel input patterns in classification problems, provided that the inputs to the classifier are real values. The reliability measures of the VI-net are implemented as additional output nodes of the underlying RBFN. Weights associated with the reliability nodes are given analytically based on training statistics from the fitting of the target function, and thus the reliability measures can be added to a standard RBFN with no additional training effort. 1.

Due to their powerful optimization property, genetic algorithms (GAs) are currently being investigated for the development of adaptive or self-tuning fuzzy logic control systems. This paper presents a neuro-fuzzy logic controller (NFLC) where all of its parameters can be tuned simultaneously by GA. The structure of the controller is based on the Radial Basis Function neural network (RBF) with Gaussian membership functions. The NFLC tuned by GA can somewhat eliminate laborious design steps such as manual tuning of the membership functions and selection of the fuzzy rules. The GA implementation incorporates dynamic crossover and mutation probabilistic rates for faster convergence. A flexible position coding strategy of the NFLC parameters is also implemented to obtain near optimal solutions. The performance of the proposed controller is compared with a conventional fuzzy controller and a PID controller tuned by GA. Simulation results show that the proposed controller offers encouraging advantages and has better performance.

This paper describes a simple method of obtaining longer-term predictions from a nonlinear time-series, assuming one already has a reasonably good short-term predictor. The usefulness of the technique is that it eliminates, to some extent, the systematic errors of the iterated short-term predictor. The technique we describe also provides an indication of the prediction horizon. We consider systems with both observational and dynamic noise and analyse a number of artificial and experimental systems obtaining consistent results. We also compare this method of longer-term prediction with ensemble prediction.

Normalisation of the basis function activations in a radial basis function (RBF) network is a common way of achieving the partition of unity often desired for modelling applications. It results in the basis functions covering the whole of the input space to the same degree. However, normalisation of the basis functions can lead to other effects which are sometimes less desireable for modelling applications. This paper describes some side effects of normalisation which fundamentally alter properties of the basis functions, e.g. the shape is no longer uniform, maxima of basis functions can be shifted from their centres, and the basis functions are no longer guaranteed to decrease monotonically as distance from their centre increases -- in many cases basis functions can re-appear far from the basis function centre. This paper examines how these phenomena occur, and analyses theoretically and experimentally the effect of normalisation on the least squares solution to the weights problem. ...

Abstract—The problem of distinguishing and classifying the responses of analog integrated circuits containing catastrophic faults has aroused recent interest. The problem is made more difficult when parametric variations are taken into account. Hence, statistical methods and techniques such as neural networks have been employed to automate classification. The major drawback to such techniques has been the implicit assumption that the variances of the responses of faulty circuits have been the same as each other and the same as that of the fault-free circuit. This assumption can be shown to be false. Neural networks, moreover, have proved to be slow. This paper describes a new neural network structure that clusters responses assuming different means and variances. Sophisticated statistical techniques are employed to handle situations where the variance tends to zero, such as happens with a fault that causes a response to be stuck at a supply rail. Two example circuits are used to show that this technique is significantly more accurate than other classification methods. A set of responses can be classified in the order of 1 s. Index Terms—Automatic test pattern generation (ATPG) testing, fault-diagnosis, quiescent supply current (IDDQ), mixed-signal_test. I.

Recently, adaptive control systems utilizing artificial intelligent techniques are being actively investigated in many applications. Neural networks with their powerful learning capability are being sought as the basis for many adaptive control systems where on-line adaptation can be implemented. Fuzzy logic on the other hand have been proven to be rather popular in many control system applications providing a rule-base like structure. In this paper, an adaptive neuro-fuzzy control system is proposed where the Radial Basis Function neural network (RBF) is implemented as a neuro-fuzzy controller (NFC) and the General Regression neural network (GRNN) as a predictor. The adaptation of the system involves three procedures as follows: (1) tuning of the control actions or rules, (2) trimming of the control actions, and (3) adjustment of the controller output gain. The tuning method is a non-gradient descent method based on the predicted system response, which is able to self-organize the control actions from an initial stage. The trimming scheme can help to reduce the aggressiveness of the particular control rules in order to stabilize the response to the set-points more effectively, while the controller gain adjustment scheme can be applied if the appropriate controller output gain is difficult to be determined heuristically. To show the effectiveness of this proposed methodology, it's performance is compared with the well known Generalized Predictive Control (GPC) technique which has a combination of both adaptive and predictive control schemes. Comparisons are made with respect to transient response, disturbance rejection and changes in plant dynamics, The proposed control system is also applied to control a single link manipulator. The results show that it exhibits robust...

In this paper, we propose a Wavelet Neural Network with Hybrid Learning Approach (WNN-HLA). A novel hybrid learning approach, which combines the on-line partition method (OLPM) and the gradient descent method, is proposed to identify a parsimonious internal structure and adjust the parameters of WNN-HLA model. First, the proposed OLPM is an online method and is a distance-based connectionist clustering method. Unlike the traditional cluster techniques that only consider the total variation to update the one mean and deviation. Second, a back-propagation learning method is used to adjust the parameters for the desired outputs. Several simulation examples have been given to illustrate the performance and effectiveness of the proposed model. The computer simulations demonstrate that the proposed WNN-HLA model performs better than some existing models.

Genetic Algorithms (GAs) with gene dependent mutation probability applied to non-stationary optimization problems are investigated in this paper. In the problems studied here, the fitness function changes during the search carried out by the GA. In the GA investigated, each gene is associated with an independent mutation probability. The knowledge obtained during the evolution is utilized to update the mutation probabilities. If the modification of a set of genes is useful when the problem changes, the mutation probabilities of these genes are increased. In this way, the search in the solution space is concentrated into regions associated with the genes with higher mutation probabilities. The class of non-stationary problems where this GA can be interesting and its limitations are investigated.