Recurrent neural networks can be used to map input sequences to output sequences, such as for recognition, production or prediction problems. However, practical difficulties have been reported in training recurrent neural networks to perform tasks in which the temporal contingencies present in the input/output sequences span long intervals. We showwhy gradient based learning algorithms face an increasingly difficult problem as the duration of the dependencies to be captured increases. These results expose a trade-off between efficient learning by gradient descent and latching on information for long periods. Based on an understanding of this problem, alternatives to standard gradient descent are considered.

Introduction We propose a complementary approach to discrete dynamical systems, using predicate transformers. We present general concepts like invariance and attraction, and we propose properties to characterize the structure of invariants. Then we present the concept of composition of dynamical systems. We define algebraic operators on systems and we try to discover how dynamical properties of small systems are preserved or transformed when these are composed into more complex systems. Finally, we illustrate this approach on an example. We work with a space E (e.g. N, or R), and extend functions from E to E into functions PE ! PE, which are invertible. Any subset of E can be described by a predicate. For example, an interval [a; b] ` R is defined by the predicate P (x) =

this paper, an immune idiotypic network in which the prevailing behaviour is oscillatory is studied in details. It is shown how connecting an elementary three-clone network in a frustrating way transforms the oscillatory regime into a chaotic one. This chaotic regime is further analysed and several interesting aspects are discussed such as the variable homogeneity, the intrinsic chaotic itinerancy among brief oscillatory regimes and the strong unpredictability. In addition, dynamical regimes obtained by frustrating the connectivity of HN and CML are presented and the similarities as well as the differences with the INN dynamics are emphasized. Common to all these networks is the description of the frustrated chaos as a succession of attempts to relax the network into one of the oscillatory regimes given by a weaker and non-frustrated connectivity, an impossible achievement making the dynamics rambling over brief but repelling orbits.

We consider a simplified neural network model for a ring of four neurons where each neuron receives two time delayed inputs: one from itself and another from the previous neuron. Local stability analysis of the positive equilibrium leads to a characteristic equation containing products of four transcendental functions. By analyzing the equivalent system of four scalar transcendental equations, we obtain sufficient conditions for the linear stability of the positive equilibrium. Furthermore, we show that a Hopf bifurcation can occur when the positive equilibrium loses stability. Key words. Neural network model, time delay, stability, Hopf bifurcation. AMS subject classifications. 34K20, 92B20. Running head. A Neural Network Model with Multiple Time Delays 1 Department of Applied Mathematics, University of Waterloo, Waterloo, Ontario, Canada N2L 3G1. 2 Department of Mathematics, Statistics, and Computing Science, Dalhousie University, Halifax, Nova Scotia, Canada B3H 3J5. 3 Depar...

Comparing to other existing approaches to deal with temporal data, the interest in recurrent networks is mostly due to their capability of implementing adaptive long-term memories. In spite of these potentialities, optimal training of parametric dynamical systems is not an easy task. In this paper we focus on sequence processing tasks such as production and classification. After reviewing some approaches proposed in the literature we describe the difficulties that are encountered in training recurrent networks and alternatives that can be pursued to circumvent these problems. In particular we consider alternative optimization techiques that can be better suited to deal with long-term dependencies, and prior knowledge injection techniques that may simplify the learning task in such situations. We finally discuss the implications that achievements in recurrent networks research might have for the technology of adaptive systems and artificial neural networks. 1 Introduction The...

We consider a ring of identical elements with time delayed, nearest neighbour coupling. The individual elements are modelled by a scalar delay differential equation which includes linear decay and nonlinear delayed feedback. The linear stability of the trivial solution is completely analyzed and illustrated in the parameter space of the coupling strength and the coupling delay. Conditions for global stability of the trivial solution are also given. The bifurcation and stability of nontrivial synchronous solutions from the trivial solution is analyzed using a centre manifold construction.