## Finite State Machines and Recurrent Neural Networks -- Automata and Dynamical Systems Approaches (1998)

Venue: | Neural Networks and Pattern Recognition |

Citations: | 29 - 11 self |

### BibTeX

@INPROCEEDINGS{Horne98finitestate,

author = {Bill G. Horne and C. Lee Giles and Pete C. Collingwood and School Of Computing and Man Sci and Peter Tino and Peter Tino},

title = {Finite State Machines and Recurrent Neural Networks -- Automata and Dynamical Systems Approaches},

booktitle = {Neural Networks and Pattern Recognition},

year = {1998},

pages = {171--220},

publisher = {Academic Press}

}

### Years of Citing Articles

### OpenURL

### Abstract

We present two approaches to the analysis of the relationship between a recurrent neural network (RNN) and the finite state machine M the network is able to exactly mimic. First, the network is treated as a state machine and the relationship between the RNN and M is established in the context of algebraic theory of automata. In the second approach, the RNN is viewed as a set of discrete-time dynamical systems associated with input symbols of M. In particular, issues concerning network representation of loops and cycles in the state transition diagram of M are shown to provide a basis for the interpretation of learning process from the point of view of bifurcation analysis. The circumstances under which a loop corresponding to an input symbol x is represented by an attractive fixed point of the underlying dynamical system associated with x are investigated. For the case of two recurrent neurons, under some assumptions on weight values, bifurcations can be understood in the geometrical c...

### Citations

1539 | Finding structure in time
- Elman
- 1990
(Show Context)
Citation Context ... the creation of a new fixed point is the saddle node bifurcation. 1 Introduction The relationship between recurrent neural networks (RNN) and automata has been treated by many [29], [26], [8], [10], =-=[15]-=-, [17], [5], [38], [39], [28], [11], [23]. State units' activations represent past histories and clusters of these activations can represent the states of the generating automaton [18]. In this contri... |

1328 |
Nonlinear oscillations, dynamical systems, and bifurcations of vector fields
- Guckenheimer, Holmes
- 1983
(Show Context)
Citation Context ...ainly an absorbing set of itself under f . B may be an attracting set of f , or it may contain an attractive set of f 7 , or none of the two 8 . To learn more about the theory of DSs, see for example =-=[19]-=-. 4 in sense of inclusion 5 B denotes the closure of B 6 Loosely speaking, strange attractors are attractive sets that are topologically distinct from (i.e. cannot be transformed by a homeomorphism to... |

590 |
Neurons with graded response have collective computational properties like those of two-state neurons
- Hopfield
- 1984
(Show Context)
Citation Context ...works assume some form of structure in the weight matrix describing connectivity pattern among recurrent neurons. For example, symmetric connectivity and absence of self-interactions enabled Hopfield =-=[22]-=- to interpret the network as a physical system having energy minima in attractive fixed points of the network. These rather strict conditions were weakened in [7], where a more easily satisfied condit... |

337 |
An introduction to chaotic dynamical systems
- DEVANEY
- 2003
(Show Context)
Citation Context ...active fixed points and periodic orbits are trivial examples of attractive sets. Much more complicated attractive sets can be found in dynamical systems literature under the name strange attractors 6 =-=[12]-=-. As in the case of an attractive fixed point, the basin of attraction of an attractive set ~ B is the set of all points whose orbits converge to ~ B. If B`A is positively invariant set of f then it i... |

311 |
Combinational profiles of sequential benchmark circuits
- Brglez, Bryan, et al.
- 1989
(Show Context)
Citation Context ...each input symbol has no arrows. As another example, Consider a FSM M in figure 24. It is a FSM taken from the database of the International Symposium on Circuits and Systems (Portland, Oregon, 1989) =-=[4]-=-. In each of its 7 states there is an a-loop with output 0 except for a-loops in states 4 and 7. The training set consists of 3500 training strings 24 of input string length 3--35 and is ordered accor... |

245 |
Attractor dynamics and parallelism in a connectionist sequential machine
- Jordan
- 1986
(Show Context)
Citation Context ...n responsible for the creation of a new fixed point is the saddle node bifurcation. 1 Introduction The relationship between recurrent neural networks (RNN) and automata has been treated by many [29], =-=[26]-=-, [8], [10], [15], [17], [5], [38], [39], [28], [11], [23]. State units' activations represent past histories and clusters of these activations can represent the states of the generating automaton [18... |

172 |
Learning and extracting finite state automata with second-order recurrent neural networks
- Giles, Miller, et al.
- 1992
(Show Context)
Citation Context ...reation of a new fixed point is the saddle node bifurcation. 1 Introduction The relationship between recurrent neural networks (RNN) and automata has been treated by many [29], [26], [8], [10], [15], =-=[17]-=-, [5], [38], [39], [28], [11], [23]. State units' activations represent past histories and clusters of these activations can represent the states of the generating automaton [18]. In this contribution... |

149 |
Finite state automata and simple recurrent networks
- Cleeremans, Servan-Schreiber, et al.
- 1989
(Show Context)
Citation Context ...onsible for the creation of a new fixed point is the saddle node bifurcation. 1 Introduction The relationship between recurrent neural networks (RNN) and automata has been treated by many [29], [26], =-=[8]-=-, [10], [15], [17], [5], [38], [39], [28], [11], [23]. State units' activations represent past histories and clusters of these activations can represent the states of the generating automaton [18]. In... |

114 |
On the dynamics of small continuous-time recurrent neural networks
- Beer
- 1995
(Show Context)
Citation Context ...tion of Dynamical Systems RNNs can be viewed as discrete-time DSs. Literature dealing with the relationship between RNNs and DSs is quite rich: [20], [3], [16], [6], [7] [24], [26], [35], [36], [34], =-=[2]-=-, [21], for example. However, as it has been already mentioned, the task of complete understanding of the global dynamical behaviour of a given DS is not at all an easy one. In [36] it is shown that n... |

98 |
Convergent activation dynamics in continuous timenetwork
- Hirsch
- 1989
(Show Context)
Citation Context ...ique introduced in [17] are supported. 6 RNN as a Collection of Dynamical Systems RNNs can be viewed as discrete-time DSs. Literature dealing with the relationship between RNNs and DSs is quite rich: =-=[20]-=-, [3], [16], [6], [7] [24], [26], [35], [36], [34], [2], [21], for example. However, as it has been already mentioned, the task of complete understanding of the global dynamical behaviour of a given D... |

82 |
Induction of finite-state languages using second-order recurrent networks
- Watrous, Kuhn
- 1992
(Show Context)
Citation Context ...a new fixed point is the saddle node bifurcation. 1 Introduction The relationship between recurrent neural networks (RNN) and automata has been treated by many [29], [26], [8], [10], [15], [17], [5], =-=[38]-=-, [39], [28], [11], [23]. State units' activations represent past histories and clusters of these activations can represent the states of the generating automaton [18]. In this contribution, the relat... |

45 |
Learning finite state machines with self-clustering recurrent networks
- Zeng, Goodman, et al.
- 1993
(Show Context)
Citation Context ...fixed point is the saddle node bifurcation. 1 Introduction The relationship between recurrent neural networks (RNN) and automata has been treated by many [29], [26], [8], [10], [15], [17], [5], [38], =-=[39]-=-, [28], [11], [23]. State units' activations represent past histories and clusters of these activations can represent the states of the generating automaton [18]. In this contribution, the relationshi... |

37 |
Efficient simulation of finite automata by neural nets
- Alon, Dewdney, et al.
- 1991
(Show Context)
Citation Context ...he language) can be quantified by the minimal number of neurons needed to recognize the language. In the context of mealy machines and threshold networks a similar problem was attacked by Alon et al. =-=[1]-=- and Horne and Hush [23]. An attempt to predict the minimal second-order RNN size so that the network can learn to accept a given regular language is presented in [33]. The predicted numbers of neuron... |

37 | Extracting and learning an unknown grammar with recurrent neural networks - Giles, Miller, et al. - 1992 |

36 |
Bounds on the complexity of recurrent neural network implementations of finite state machines
- Horne, Hush
- 1996
(Show Context)
Citation Context ... saddle node bifurcation. 1 Introduction The relationship between recurrent neural networks (RNN) and automata has been treated by many [29], [26], [8], [10], [15], [17], [5], [38], [39], [28], [11], =-=[23]-=-. State units' activations represent past histories and clusters of these activations can represent the states of the generating automaton [18]. In this contribution, the relationship between a RNN an... |

32 | A unified gradient-descent/clustering architecture for finite state machine induction
- Das, Mozer
- 1994
(Show Context)
Citation Context ...is the saddle node bifurcation. 1 Introduction The relationship between recurrent neural networks (RNN) and automata has been treated by many [29], [26], [8], [10], [15], [17], [5], [38], [39], [28], =-=[11]-=-, [23]. State units' activations represent past histories and clusters of these activations can represent the states of the generating automaton [18]. In this contribution, the relationship between a ... |

30 |
First order recurrent neural networks and deterministic finite state automata
- Manolios, Fanelli
- 1994
(Show Context)
Citation Context ...point is the saddle node bifurcation. 1 Introduction The relationship between recurrent neural networks (RNN) and automata has been treated by many [29], [26], [8], [10], [15], [17], [5], [38], [39], =-=[28]-=-, [11], [23]. State units' activations represent past histories and clusters of these activations can represent the states of the generating automaton [18]. In this contribution, the relationship betw... |

29 | Period-Doublings to Chaos in a Simple Neural Network: An Analytic Proof
- Wang
- 1991
(Show Context)
Citation Context ... 6 RNN as a Collection of Dynamical Systems RNNs can be viewed as discrete-time DSs. Literature dealing with the relationship between RNNs and DSs is quite rich: [20], [3], [16], [6], [7] [24], [26], =-=[35]-=-, [36], [34], [2], [21], for example. However, as it has been already mentioned, the task of complete understanding of the global dynamical behaviour of a given DS is not at all an easy one. In [36] i... |

24 | Bifurcations in the learning of recurrent neural networks
- Doya
- 1992
(Show Context)
Citation Context ... Before training, the connection weights are set to small random values and as a consequence, the network has only one attractor basin. This implies that the network must undergo several bifurcations =-=[13]-=-. This can have an undesirable effect on the training process, since the gradient descent learning may get into trouble. At bifurcations points, the output of a network can change discontinuously with... |

22 |
Linear-Input Logic
- Minnick
- 1961
(Show Context)
Citation Context ...rcation responsible for the creation of a new fixed point is the saddle node bifurcation. 1 Introduction The relationship between recurrent neural networks (RNN) and automata has been treated by many =-=[29]-=-, [26], [8], [10], [15], [17], [5], [38], [39], [28], [11], [23]. State units' activations represent past histories and clusters of these activations can represent the states of the generating automat... |

22 |
Learning and extracting initial mealy machines with a modular neural network model
- Tino, Sajda
- 1995
(Show Context)
Citation Context ...ymbol x, gradually degradate upon repeated presentation of x. Sections 2 and 3 bring brief introductions to state machines and dynamical systems respectively. Section 4 is devoted to the model of RNN =-=[31]-=- used for learning FSMs. 2 State Machines This section introduces the concept of state machine, which is a generalized finite state machine with possibly uncountable number of states. When viewed as a... |

20 |
Induction of finite-state automata using second-order recurrent networks
- Watrous, Kuhn
- 1992
(Show Context)
Citation Context ... particular �� m x ((q j ) N ) ` (q j ) N ` " i0 (�� im x ) \Gamma1 (N y j x ); j = 1; :::; m: (13) Some researchers attempted to extract learned automaton from a trained recurrent networ=-=k [17], [8], [37]-=-, [31]. Extraction procedures rely on the assumption that equivalent network states are grouped together in well-separated regions in the recurrent neurons' activation space. After training, the netwo... |

17 |
Stability of fixed points and periodic orbits and bifurcations in analog neural networks. Neural Networks
- Blum, Wang
- 1992
(Show Context)
Citation Context ...ntroduced in [17] are supported. 6 RNN as a Collection of Dynamical Systems RNNs can be viewed as discrete-time DSs. Literature dealing with the relationship between RNNs and DSs is quite rich: [20], =-=[3]-=-, [16], [6], [7] [24], [26], [35], [36], [34], [2], [21], for example. However, as it has been already mentioned, the task of complete understanding of the global dynamical behaviour of a given DS is ... |

13 | Learning context-free grammars: capabilities and limitations of a recurrent neural network with an external memory stack
- Das, Giles, et al.
- 1992
(Show Context)
Citation Context ...le for the creation of a new fixed point is the saddle node bifurcation. 1 Introduction The relationship between recurrent neural networks (RNN) and automata has been treated by many [29], [26], [8], =-=[10]-=-, [15], [17], [5], [38], [39], [28], [11], [23]. State units' activations represent past histories and clusters of these activations can represent the states of the generating automaton [18]. In this ... |

13 |
The complexity of language recognition by neural networks, in
- Siegelmann, Sontag, et al.
- 1992
(Show Context)
Citation Context ... One of the most promising neural acceptors of regular languages [32] is the second-order RNN introduced by Giles et al. [17]. However, the practical aspects of the acceptance issue are still unclear =-=[33]-=-. The difficulty of acceptance of a given language by a neural network (the neural complexity of the language) can be quantified by the minimal number of neurons needed to recognize the language. In t... |

10 |
Location and Stability of the High-Gain Equilibria of Nonlinear Neural Networks
- Vidyasagar
- 1993
(Show Context)
Citation Context ...Collection of Dynamical Systems RNNs can be viewed as discrete-time DSs. Literature dealing with the relationship between RNNs and DSs is quite rich: [20], [3], [16], [6], [7] [24], [26], [35], [36], =-=[34]-=-, [2], [21], for example. However, as it has been already mentioned, the task of complete understanding of the global dynamical behaviour of a given DS is not at all an easy one. In [36] it is shown t... |

7 | Bifurcations of Recurrent Neural Networks in Gradient Descent Learning
- Doya
- 1993
(Show Context)
Citation Context ...y get into trouble. At bifurcations points, the output of a network can change discontinuously with the change of parameters and therefore convergence of gradient descent algorithms is not guaranteed =-=[14]-=-. In the following a possible application of these ideas to the problem of determination of the complexity of language recognition by neural networks will be discussed briefly. Any FSM with binary out... |

7 |
at high gain in discrete time recurrent networks
- Hirsch
- 1994
(Show Context)
Citation Context ...of Dynamical Systems RNNs can be viewed as discrete-time DSs. Literature dealing with the relationship between RNNs and DSs is quite rich: [20], [3], [16], [6], [7] [24], [26], [35], [36], [34], [2], =-=[21]-=-, for example. However, as it has been already mentioned, the task of complete understanding of the global dynamical behaviour of a given DS is not at all an easy one. In [36] it is shown that network... |

6 |
Computation in discrete-time dynamical systems
- Casey
- 1995
(Show Context)
Citation Context ...ere is a vertex v 2 f0; 1g L such that v is in the closure of (q) N , the loop is likely to be represented by an attractive fixed point 1 "near" v. A related work was independently done by C=-=asey [5], [6]-=-. In his setting, RNN is assumed to operate in a noisy environment (representing for example a noise corresponding to round-off errors in computations performed on a digital computer). RNNs are traine... |

6 |
Absolute stability conditions for discrete-time recurrent neural networks
- Jin, Nikiforuk, et al.
- 1994
(Show Context)
Citation Context ...pecial types. In case of two recurrent neurons with sigmoidal activation function g, they give results for weight matrices with diagonal elements equal to zero 17 . Recently, Jin, Nikifiruk and Gupta =-=[25]-=- reported new results on the absolute stability for a rather general class of recurrent neural networks. Conditions under which all fixed points of the network are attractive were determined by the we... |

5 | Representation of temporal patterns in recurrent networks
- Cummins
- 1993
(Show Context)
Citation Context ...attractor basins to store distinct internal states. The network solves the task of FSM simulation by location of point and periodic attractors and the shaping of their respective basins of attraction =-=[9]-=-. Before training, the connection weights are set to small random values and as a consequence, the network has only one attractor basin. This implies that the network must undergo several bifurcations... |

5 |
Global dynamics in neural networks
- Franklin, Garzon
- 1989
(Show Context)
Citation Context ...uced in [17] are supported. 6 RNN as a Collection of Dynamical Systems RNNs can be viewed as discrete-time DSs. Literature dealing with the relationship between RNNs and DSs is quite rich: [20], [3], =-=[16]-=-, [6], [7] [24], [26], [35], [36], [34], [2], [21], for example. However, as it has been already mentioned, the task of complete understanding of the global dynamical behaviour of a given DS is not at... |

5 |
An Introduction to Automata Theory
- Shields
- 1987
(Show Context)
Citation Context ... max = 100. of word's last symbol is 1. Hence, the network output is used to decide whether a word does belong to the language, or not. One of the most promising neural acceptors of regular languages =-=[32]-=- is the second-order RNN introduced by Giles et al. [17]. However, the practical aspects of the acceptance issue are still unclear [33]. The difficulty of acceptance of a given language by a neural ne... |

4 | the symmetric weight condition for convergent dynamics in discrete-time recurrent networks
- Casey
- 1995
(Show Context)
Citation Context ...] are supported. 6 RNN as a Collection of Dynamical Systems RNNs can be viewed as discrete-time DSs. Literature dealing with the relationship between RNNs and DSs is quite rich: [20], [3], [16], [6], =-=[7]-=- [24], [26], [35], [36], [34], [2], [21], for example. However, as it has been already mentioned, the task of complete understanding of the global dynamical behaviour of a given DS is not at all an ea... |

4 |
Discrete-Time versus Continuous-Time Models of Neural Networks
- Wang, Blum
- 1992
(Show Context)
Citation Context ... as a Collection of Dynamical Systems RNNs can be viewed as discrete-time DSs. Literature dealing with the relationship between RNNs and DSs is quite rich: [20], [3], [16], [6], [7] [24], [26], [35], =-=[36]-=-, [34], [2], [21], for example. However, as it has been already mentioned, the task of complete understanding of the global dynamical behaviour of a given DS is not at all an easy one. In [36] it is s... |

3 |
analysis of the brain-state-in-a-box neural models
- Hui, Zak
- 1992
(Show Context)
Citation Context ...e supported. 6 RNN as a Collection of Dynamical Systems RNNs can be viewed as discrete-time DSs. Literature dealing with the relationship between RNNs and DSs is quite rich: [20], [3], [16], [6], [7] =-=[24]-=-, [26], [35], [36], [34], [2], [21], for example. However, as it has been already mentioned, the task of complete understanding of the global dynamical behaviour of a given DS is not at all an easy on... |

3 |
Non-standard topologies of neuron field in self-organizing feature maps
- Tino, Jelly, et al.
- 1994
(Show Context)
Citation Context ...ns found in vector-coding algorithms using independent cluster centers, while avoiding time consuming approximation of input space topology typical of classical regular-grid topologies of Kohonen Map =-=[30]. Ot-=-her approaches to RNN state space clustering are discussed in [31]. Having approximated the regions (q) N , the automaton �� N 1 is constructed via determining arcs in the corresponding transition... |

2 |
Computation Dynamics in Discrete-Time Recurrent Neural Networks
- Casey
- 1993
(Show Context)
Citation Context ...n of a new fixed point is the saddle node bifurcation. 1 Introduction The relationship between recurrent neural networks (RNN) and automata has been treated by many [29], [26], [8], [10], [15], [17], =-=[5]-=-, [38], [39], [28], [11], [23]. State units' activations represent past histories and clusters of these activations can represent the states of the generating automaton [18]. In this contribution, the... |

2 |
Fixed point analysis for discrete-time recurrent neural networks
- Li
- 1992
(Show Context)
Citation Context ... The map �� x is transformed to the map �� �� x by multiplying weights W iln by a scalar �� ? 0, i.e. �� �� x (s) = �� x (��s). �� is also called the neuron gain. T=-=he following Lemma was proved by Li [27]. It is stated for map-=-s �� x and accommodated with our notation. It tells us under what conditions one may expect an attractive fixed point of �� �� x to exist "near" a vertex v 2 f0; 1g L . Lemma 1: ... |