A nonlinear black box structure for a dynamical system is a model structure that is prepared to describe virtually any nonlinear dynamics. There has been considerable recent interest in this area with structures based on neural networks, radial basis networks, wavelet networks, hinging hyperplanes, as well as wavelet transform based methods and models based on fuzzy sets and fuzzy rules. This paper describes all these approaches in a common framework, from a user's perspective. It focuses on what are the common features in the different approaches, the choices that have to be made and what considerations are relevant for a successful system identification application of these techniques. It is pointed out that the nonlinear structures can be seen as a concatenation of a mapping from observed data to a regression vector and a nonlinear mapping from the regressor space to the output space. These mappings are discussed separately. The latter mapping is usually formed as a basis function e...

It has recently been shown that gradient-descent learning algorithms for recurrent neural networks can perform poorly on tasks that involve long--term dependencies, i.e. those problems for which the desired output depends on inputs presented at times far in the past. We show tht the long--term dependencies problem is lessened for a class of architectures called NARX recurrent neural networks, which have powerful representational capabilities. We have previously reported that gradient descent learning can be more effective in NARX networks than in recurrent neural network architectures that have "hidden states" on problems including grammatical inference and nonlinear system identification. Typically, the network converges much faster and generalizes better than other networks. The results in this paper are consistent with this phenomenon. We present some experimental results which show that NARX networks can often retain information for two to three times as long as conventi...

Recently, fully connected recurrent neural networks have been proven to be computationally rich --- at least as powerful as Turing machines. This work focuses on another network which is popular in control applications and has been found to be very effective at learning a variety of problems. These networks are based upon Nonlinear AutoRegressive models with eXogenous Inputs (NARX models), and are therefore called NARX networks. As opposed to other recurrent networks, NARX networks have a limited feedback which comes only from the output neuron rather than from hidden states. They are formalized by y(t) = \Psi i u(t \Gamma nu ); : : : ; u(t \Gamma 1); u(t); y(t \Gamma ny ); : : : ; y(t \Gamma 1) j ; where u(t) and y(t) represent input and output of the network at time t, nu and ny are the input and output order, and the function \Psi is the mapping performed by a Multilayer Perceptron. We constructively prove that the NARX networks with a finite number of parameters are computation...

: This study investigates the properties of arti#cial recurrent neural networks. Particular attention is paid to the question of how these nets can be applied to the identi#cation and control of non-linear dynamic processes. Since these kind of processes can only insu#ciently be modelled by conventional methods, di#erent approaches are required. Neural networks are considered to be useful for this purpose due to their ability to approximate a wide class of continuous functions. Among the numerous network structures, especially the recurrentmulti-layer perceptron #RMLP# architecture is promising from application point of view. This network architecture has the wellknown properties of multi layer perceptrons and moreover these nets have the ability to incorporate temporal behavior. Departing from the original process description the applicability of RMLPs is investigated and di#erent learning algorithms for this network class are outlined. Furthermore, besides the conventional...

Emnet for n rv rende licentiatafhandling er design af neurale netv rks ltre. Filtre baseret pa neurale netv rk kan ses som udvidelser af det klassiske line re adaptive l-ter rettet mod modellering af uline re sammenh nge. Hovedv gten l gges pa en neural netv rks implementering af den ikke-rekursive, uline re adaptive model med additiv st j. Formalet er at klarl gge en r kke faser forbundet med design af neural netv rks arkitekturer med henblik pa at udf re forskellige \black-box " modellerings opgaver sa som: System identi kation, invers modellering og pr diktion af tidsserier. De v senligste bidrag omfatter: Formulering af en neural netv rks baseret kanonisk lter repr sentation, der danner baggrund for udvikling af et arkitektur klassi kationssystem. I hovedsagen drejer det sig om en skelnen mellem globale og lokale modeller. Dette leder til at en r kke kendte neurale netv rks arkitekturer kan klassi ceres, og yderligere abnes der mulighed for udvikling af helt nye strukturer. I denne sammenh ng ndes en gennemgang af en r kke velkendte arkitekturer. I s rdeleshed l gges der v gt pa behandlingen af multi-lags perceptron neural netv rket.

It has recently been shown that gradient descent learning algorithms for recurrent neural networks can perform poorly on tasks that involve long--term dependencies, i.e. those problems for which the desired output depends on inputs presented at times far in the past. In this paper we explore the long--term dependencies problem for a class of architectures called NARX recurrent neural networks, which have powerful representational capabilities. We have previously reported that gradient descent learning is more effective in NARX networks than in recurrent neural network architectures that have "hidden states" on problems including grammatical inference and nonlinear system identification. Typically, the network converges much faster and generalizes better than other networks. The results in this paper are an attempt to explain this phenomenon. We present some experimental results which show that NARX networks can often retain information for two to three times as long as conventional rec...

The continuous-time version of the Local Model Network (LMN) approach of Johansen and Foss is discussed. The distinction is made between local state and global state LMNs; the former turns out to be a form of recurrent net, the latter a form of feedforward net. It is shown how local-state LMN representations of nonlinear systems can be used to design LMN-based controllers via both state observer /feedback It is suggested that this approach provides an alternative foundation for non-linear self-tuning control. 1 INTRODUCTION Control system design is based on a model of the controlled system. Such design is simplified if the model has a number of desirable attributes; it should be: 1. an accurate model of the system capturing the essential features of the actual system; 2. a robust model in the sense that small perturbations in model parameters lead to small errors between system and model states and outputs; 3. of a form lending itself to the design method being used; 4. of a form l...

Learning long-term temporal dependencies with recurrent neural networks can be a difficult problem. It has recently been shown that a class of recurrent neural networks called NARX networks perform much better than conventional recurrent neural networks for learning certain simple long-term dependency problems. The intuitive explanation for this behavior is that the output memories of a NARX network can be manifested as jump-ahead connections in the time-unfolded network. These jump-ahead connections can propagate gradient information more efficiently, thus reducing the sensitivity of the network to long-term dependencies. This work gives empirical justification to our hypothesis that similar improvements in learning long-term dependencies can be achieved with other classes of recurrent neural network architectures simply by increasing the order of the embedded memory. In particular we explore the impact of learning simple long-term dependency problems on three classes of recurrent neu...

Abstract-In this paper, back propagation is reinvestigated for an efficient evaluation of the gradient in arbitrary interconnec-tions of recurrent subsystems. It is shown that the error has to be back-propagated through the adjoint model of the system and that the gradient can only be obtained after a delay. A faster version, accelerated back propagation, that eliminates this delay, is also developed. Various schemes including the sensitivity method are studied to update the weights of the network using these gradients. Motivated by the Lyapunov approach and the adjoint model, the predictive back propagation and its variant, targeted back propagation, are proposed. A further refinement, predictive back propagation with filtering is then developed, where the states of the model are also updated. The convergence of this scheme is assured. It is shown that it is sufficient to back propagate as many time steps as the order of the system for convergence. As a preamble, convergence of on-line batch and sample-wise updates in feedforward models is analyzed using the Lyapunov approach. I.

Neural network black-box modeling is usually performed using nonlinear inputoutput models. The goal of this paper is to show that there are advantages in using nonlinear state-space models, which constitute a larger class of nonlinear dynamical models, and their corresponding state-space neural predictors. We recall the fundamentals of both input-output and state-space black-box modeling, and show the state-space neural networks to be potentially more efficient and more parsimonious than their conventional input-output counterparts. This is examplified on simulated processes as well as on a real one, the hydraulic actuator of a robot arm. 1. Introduction During the past few years, several authors [Narendra and Parthasarathy 1990, Nerrand et al. 1994] have suggested the use of neural networks for the black-box modeling of nonlinear dynamical systems. The problem of designing a mathematical model of a process using only observed data has attracted much attention, both from an academic a...