Results 1  10
of
60
Learning longterm dependencies in NARX recurrent neural networks
, 1996
"... It has recently been shown that gradientdescent learning algorithms for recurrent neural networks can perform poorly on tasks that involve longterm dependencies, i.e. those problems for which the desired output depends on inputs presented at times far in the past. We show tht the longterm de ..."
Abstract

Cited by 47 (5 self)
 Add to MetaCart
It has recently been shown that gradientdescent learning algorithms for recurrent neural networks can perform poorly on tasks that involve longterm dependencies, i.e. those problems for which the desired output depends on inputs presented at times far in the past. We show tht the longterm dependencies problem is lessened for a class of architectures called NARX recurrent neural networks, which have powerful representational capabilities. We have previously reported that gradient descent learning can be more effective in NARX networks than in recurrent neural network architectures that have "hidden states" on problems including grammatical inference and nonlinear system identification. Typically, the network converges much faster and generalizes better than other networks. The results in this paper are consistent with this phenomenon. We present some experimental results which show that NARX networks can often retain information for two to three times as long as conventi...
Computational Capabilities of Recurrent NARX Neural Networks
 IEEE Trans. on Systems, Man and Cybernetics
, 1997
"... Abstract—Recently, fully connected recurrent neural networks have been proven to be computationally rich—at least as powerful as Turing machines. This work focuses on another network which is popular in control applications and has been found to be very effective at learning a variety of problems. T ..."
Abstract

Cited by 31 (8 self)
 Add to MetaCart
Abstract—Recently, fully connected recurrent neural networks have been proven to be computationally rich—at least as powerful as Turing machines. This work focuses on another network which is popular in control applications and has been found to be very effective at learning a variety of problems. These networks are based upon Nonlinear AutoRegressive models with eXogenous Inputs (NARX models), and are therefore called NARX networks. As opposed to other recurrent networks, NARX networks have a limited feedback which comes only from the output neuron rather than from hidden states. They are formalized by y(t) =9(u(t0nu);111;u(t01); u(t);y(t0ny);111;y(t01)) where u(t) and y(t) represent input and output of the network at time t, nu and ny are the input and output order, and the function 9 is the mapping performed by a Multilayer Perceptron. We constructively prove that the NARX networks with a finite number of parameters are computationally as strong as fully connected recurrent networks and thus Turing machines. We conclude that in theory one can use the NARX models, rather than conventional recurrent networks without any computational loss even though their feedback is limited. Furthermore, these results raise the issue of what amount of feedback or recurrence is necessary for any network to be Turing equivalent and what restrictions on feedback limit computational power. I.
Digital Audio Restoration
 Applications of Digital Signal Processing to Audio and Acoustics
, 1997
"... This chapter is concerned with the application of modern signal processing techniques to the restoration of degraded audio signals. Although attention is focussed on gramophone recordings, film sound tracks and tape recordings, many of the techniques discussed have applications in other areas where ..."
Abstract

Cited by 28 (10 self)
 Add to MetaCart
This chapter is concerned with the application of modern signal processing techniques to the restoration of degraded audio signals. Although attention is focussed on gramophone recordings, film sound tracks and tape recordings, many of the techniques discussed have applications in other areas where degraded audio signals occur, such as speech transmission, telephony and hearing aids. We aim to provide a wide coverage of existing methodology while giving insight into current areas of research and future trends. 1 Introduction The introduction of high quality digital audio media such as Compact Disk (CD) and Digital Audio Tape (DAT) has dramatically raised general awareness and expectations about sound quality in all types of recordings. This, combined with an upsurge in interest in historical and nostalgic material, has led to a growing requirement for restoration of degraded sources ranging from the earliest recordings made on wax cylinders in the nineteenth century, through disc reco...
Design of Neural Network Filters
 Electronics Institute, Technical University of Denmark
, 1993
"... Emnet for n rv rende licentiatafhandling er design af neurale netv rks ltre. Filtre baseret pa neurale netv rk kan ses som udvidelser af det klassiske line re adaptive lter rettet mod modellering af uline re sammenh nge. Hovedv gten l gges pa en neural netv rks implementering af den ikkerekursive, ..."
Abstract

Cited by 21 (12 self)
 Add to MetaCart
Emnet for n rv rende licentiatafhandling er design af neurale netv rks ltre. Filtre baseret pa neurale netv rk kan ses som udvidelser af det klassiske line re adaptive lter rettet mod modellering af uline re sammenh nge. Hovedv gten l gges pa en neural netv rks implementering af den ikkerekursive, uline re adaptive model med additiv st j. Formalet er at klarl gge en r kke faser forbundet med design af neural netv rks arkitekturer med henblik pa at udf re forskellige \blackbox " modellerings opgaver sa som: System identi kation, invers modellering og pr diktion af tidsserier. De v senligste bidrag omfatter: Formulering af en neural netv rks baseret kanonisk lter repr sentation, der danner baggrund for udvikling af et arkitektur klassi kationssystem. I hovedsagen drejer det sig om en skelnen mellem globale og lokale modeller. Dette leder til at en r kke kendte neurale netv rks arkitekturer kan klassi ceres, og yderligere abnes der mulighed for udvikling af helt nye strukturer. I denne sammenh ng ndes en gennemgang af en r kke velkendte arkitekturer. I s rdeleshed l gges der v gt pa behandlingen af multilags perceptron neural netv rket.
Genetic Algorithms In Control Systems Engineering
 In Proceedings of the 12th IFAC World Congress
, 2001
"... Genetic algorithms (GAs) are global, parallel, stochastic search methods, founded on Darwinian evolutionary principles. Many variations exist, including genetic programming and multiobj ective algorithms. During the last decade GAs have been applied in a variety of areas, with varying degrees of suc ..."
Abstract

Cited by 21 (2 self)
 Add to MetaCart
Genetic algorithms (GAs) are global, parallel, stochastic search methods, founded on Darwinian evolutionary principles. Many variations exist, including genetic programming and multiobj ective algorithms. During the last decade GAs have been applied in a variety of areas, with varying degrees of success within each. A significant contribution has been made within control systems engineering. GAs exhibit considerable robustness in problem domains that are not conducive to formal, rigorous, classical analysis. They are not limited by typical control problem attributes such as illbehaved objective functions, the existence of constraints, and variations in the nature of control variables. GA software tools are available, but there is no 'industry standard'. The computational complexity of the GA has proved to be the chief impediment to realtime application of the technique. Hence, the majority of applications that use GAs are, by nature, offline. GAs have been used to optimise both structure and parameter values for both controllers and plant models. They have also been applied to fault diagnosis, stability analysis, robot pathplanning, and combinatorial problems (such as scheduling and binpacking). Hybrid approaches have proved popular, with GAs being integrated in fuzzy logic and neural computing schemes. The GA has been used as the populationbased engine for multiobjective optimisers. Multiple, Paretooptimal, solutions can be represented simultaneously. In such schemes, a decisionmaker can lead the direction of future search. Interesting future developments are anticipated in online applications and multiobjective search and decisionmaking.
Learning longterm dependencies is not as difficult with NARX recurrent neural networks
, 1996
"... It has recently been shown that gradient descent learning algorithms for recurrent neural networks can perform poorly on tasks that involve longterm dependencies, i.e. those problems for which the desired output depends on inputs presented at times far in the past. In this paper we explore the lon ..."
Abstract

Cited by 18 (3 self)
 Add to MetaCart
It has recently been shown that gradient descent learning algorithms for recurrent neural networks can perform poorly on tasks that involve longterm dependencies, i.e. those problems for which the desired output depends on inputs presented at times far in the past. In this paper we explore the longterm dependencies problem for a class of architectures called NARX recurrent neural networks, which have powerful representational capabilities. We have previously reported that gradient descent learning is more effective in NARX networks than in recurrent neural network architectures that have "hidden states" on problems including grammatical inference and nonlinear system identification. Typically, the network converges much faster and generalizes better than other networks. The results in this paper are an attempt to explain this phenomenon. We present some experimental results which show that NARX networks can often retain information for two to three times as long as conventional rec...
Identification of Nonlinear Systems using Empirical Data and Prior Knowledge  An Optimization Approach
 Automatica
, 1996
"... The choice of a parametric model structure in empirical and semiempirical nonlinear modeling is usually viewed as an important and critical step. However, it is known that by augmenting the leastsquares identification criterion with a term that imposes penalty on the nonsmoothness of the model, ..."
Abstract

Cited by 16 (5 self)
 Add to MetaCart
The choice of a parametric model structure in empirical and semiempirical nonlinear modeling is usually viewed as an important and critical step. However, it is known that by augmenting the leastsquares identification criterion with a term that imposes penalty on the nonsmoothness of the model, an optimal nonparametric model can be found explicitly. The optimal nonparametric model will depend on the particular form of the penalty, which can be looked upon as a priori knowledge, or the desired properties of the model. In this paper these results are extended in several directions, i) we show how useful types of prior knowledge other than smoothness can be included as a term in the criterion or as a constraint, and how this influences the optimal model, ii) dynamic models and a general prediction error penalty are considered, iii) we present a practical numerical procedure for the identification of a close to optimal semiparametric model. The numerical approach is motivated by the...
Algebraic Differential Equations And Rational Control Systems
 SIAM JOURNAL ON CONTROL AND OPTIMIZATION
"... An equivalence is shown between realizability of input/output operators by rational control systems and high order algebraic differential equations for input/output pairs. This generalizes, to nonlinear systems, the equivalence between autoregressive representations and finite dimensional linear rea ..."
Abstract

Cited by 15 (3 self)
 Add to MetaCart
An equivalence is shown between realizability of input/output operators by rational control systems and high order algebraic differential equations for input/output pairs. This generalizes, to nonlinear systems, the equivalence between autoregressive representations and finite dimensional linear realizability.
Multiobjective Genetic Programming: A Nonlinear System Identification Application
, 1997
"... Genetic programming (GP) is applied to a multobjective optimisation problem and the advantages of its hierarchical tree encoding scheme are compared with an earlier use of a subset representation approach which used stringencoded genetic algorithms. The GP approach is applied to the identif ..."
Abstract

Cited by 15 (4 self)
 Add to MetaCart
Genetic programming (GP) is applied to a multobjective optimisation problem and the advantages of its hierarchical tree encoding scheme are compared with an earlier use of a subset representation approach which used stringencoded genetic algorithms. The GP approach is applied to the identification of nonlinear system polynomial models and provides a tradeoff between the complexity and the performance of the models. Multiobjective Genetic Programming is not only presented as a tool for optimising multiple objectives of the problem under investigation, but also provides a means of controlling the size of tree structures used in the search process. 1. INTRODUCTION Conventional genetic programming (GP), and evolutionary algorithms (EA), in general, assign a single performance measure to each individual based on the evaluation of a scalar fitness function. However, these populationbased methods possess the characteristic of simultaneously searching for multiple solut...
The local paradigm for modeling and control: from neurofuzzy . . .
, 2001
"... The composition of simple local models for approximating complex nonlinear mappings is a common practice in recent modeling and control literature. This paper presents a comparative analysis of two different local approaches: the neurofuzzy inference system and the lazy learning approach. Neurofu ..."
Abstract

Cited by 15 (6 self)
 Add to MetaCart
The composition of simple local models for approximating complex nonlinear mappings is a common practice in recent modeling and control literature. This paper presents a comparative analysis of two different local approaches: the neurofuzzy inference system and the lazy learning approach. Neurofuzzy is a hybrid representation which combines the linguistic description typical of fuzzy inference systems, with learning procedures inspired by neural networks. Lazy learning is a memorybased technique that uses a querybased approach to select the best local model configuration by assessing and comparing different alternatives in crossvalidation. In this paper, the two approaches are compared both as learning algorithms, and as identification modules of an adaptive control system. We show that lazy learning is able to provide better modeling accuracy and higher control performance at the cost of a reduced readability of the resulting approximator. Illustrative examples of identi cation and control of a nonlinear system starting from simulated data are given.