Results 1  10
of
31
Constructing Deterministic FiniteState Automata in Recurrent Neural Networks
 Journal of the ACM
, 1996
"... Recurrent neural networks that are trained to behave like deterministic finitestate automata (DFAs) can show deteriorating performance when tested on long strings. This deteriorating performance can be attributed to the instability of the internal representation of the learned DFA states. The use o ..."
Abstract

Cited by 70 (16 self)
 Add to MetaCart
Recurrent neural networks that are trained to behave like deterministic finitestate automata (DFAs) can show deteriorating performance when tested on long strings. This deteriorating performance can be attributed to the instability of the internal representation of the learned DFA states. The use of a sigmoidal discriminant function together with the recurrent structure contribute to this instability. We prove that a simple algorithm can construct secondorder recurrent neural networks with a sparse interconnection topology and sigmoidal discriminant function such that the internal DFA state representations are stable, i.e. the constructed network correctly classifies strings of arbitrary length. The algorithm is based on encoding strengths of weights directly into the neural network. We derive a relationship between the weight strength and the number of DFA states for robust string classification. For a DFA with n states and m input alphabet symbols, the constructive algorithm genera...
Computational Capabilities of Recurrent NARX Neural Networks
 IEEE Trans. on Systems, Man and Cybernetics
, 1997
"... Abstract—Recently, fully connected recurrent neural networks have been proven to be computationally rich—at least as powerful as Turing machines. This work focuses on another network which is popular in control applications and has been found to be very effective at learning a variety of problems. T ..."
Abstract

Cited by 31 (8 self)
 Add to MetaCart
Abstract—Recently, fully connected recurrent neural networks have been proven to be computationally rich—at least as powerful as Turing machines. This work focuses on another network which is popular in control applications and has been found to be very effective at learning a variety of problems. These networks are based upon Nonlinear AutoRegressive models with eXogenous Inputs (NARX models), and are therefore called NARX networks. As opposed to other recurrent networks, NARX networks have a limited feedback which comes only from the output neuron rather than from hidden states. They are formalized by y(t) =9(u(t0nu);111;u(t01); u(t);y(t0ny);111;y(t01)) where u(t) and y(t) represent input and output of the network at time t, nu and ny are the input and output order, and the function 9 is the mapping performed by a Multilayer Perceptron. We constructively prove that the NARX networks with a finite number of parameters are computationally as strong as fully connected recurrent networks and thus Turing machines. We conclude that in theory one can use the NARX models, rather than conventional recurrent networks without any computational loss even though their feedback is limited. Furthermore, these results raise the issue of what amount of feedback or recurrence is necessary for any network to be Turing equivalent and what restrictions on feedback limit computational power. I.
Finite State Machines and Recurrent Neural Networks  Automata and Dynamical Systems Approaches
 Neural Networks and Pattern Recognition
, 1998
"... We present two approaches to the analysis of the relationship between a recurrent neural network (RNN) and the finite state machine M the network is able to exactly mimic. First, the network is treated as a state machine and the relationship between the RNN and M is established in the context of alg ..."
Abstract

Cited by 29 (11 self)
 Add to MetaCart
We present two approaches to the analysis of the relationship between a recurrent neural network (RNN) and the finite state machine M the network is able to exactly mimic. First, the network is treated as a state machine and the relationship between the RNN and M is established in the context of algebraic theory of automata. In the second approach, the RNN is viewed as a set of discretetime dynamical systems associated with input symbols of M. In particular, issues concerning network representation of loops and cycles in the state transition diagram of M are shown to provide a basis for the interpretation of learning process from the point of view of bifurcation analysis. The circumstances under which a loop corresponding to an input symbol x is represented by an attractive fixed point of the underlying dynamical system associated with x are investigated. For the case of two recurrent neurons, under some assumptions on weight values, bifurcations can be understood in the geometrical c...
Rule Extraction from Recurrent Neural Networks: a Taxonomy and Review
 Neural Computation
, 2005
"... this paper, the progress of this development is reviewed and analysed in detail. In order to structure the survey and to evaluate the techniques, a taxonomy, specifically designed for this purpose, has been developed. Moreover, important open research issues are identified, that, if addressed pr ..."
Abstract

Cited by 24 (3 self)
 Add to MetaCart
this paper, the progress of this development is reviewed and analysed in detail. In order to structure the survey and to evaluate the techniques, a taxonomy, specifically designed for this purpose, has been developed. Moreover, important open research issues are identified, that, if addressed properly, possibly can give the field a significant push forward
Stable Encoding of Large FiniteState Automata in Recurrent Neural Networks with Sigmoid Discriminants
 Neural Computation
, 1996
"... We propose an algorithm for encoding deterministic finitestate automata (DFAs) in secondorder recurrent neural networks with sigmoidal discriminant function and we prove that the languages accepted by the constructed network and the DFA are identical. The desired finitestate network dynamics is a ..."
Abstract

Cited by 21 (9 self)
 Add to MetaCart
We propose an algorithm for encoding deterministic finitestate automata (DFAs) in secondorder recurrent neural networks with sigmoidal discriminant function and we prove that the languages accepted by the constructed network and the DFA are identical. The desired finitestate network dynamics is achieved by programming a small subset of all weights. A worst case analysis reveals a relationship between the weight strength and the maximum allowed network size which guarantees finitestate behavior of the constructed network. We illustrate the method by encoding random DFAs with 10, 100, and 1,000 states. While the theory predicts that the weight strength scales with the DFA size, we find the weight strength to be almost constant for all the experiments. These results can be explained by noting that the generated DFAs represent average cases. We empirically demonstrate the existence of extreme DFAs for which the weight strength scales with DFA size. 1 INTRODUCTION It is possible to tra...
Stable Encoding of FiniteState Machines in DiscreteTime Recurrent Neural Nets with Sigmoid Units
, 1998
"... In recent years, there has been a lot of interest in the use of discretetime recurrent neural nets (DTRNN) to learn finitestate tasks, with interesting results regarding the induction of simple finitestate machines from inputoutput strings. Parallel work has studied the computational power of DT ..."
Abstract

Cited by 14 (3 self)
 Add to MetaCart
In recent years, there has been a lot of interest in the use of discretetime recurrent neural nets (DTRNN) to learn finitestate tasks, with interesting results regarding the induction of simple finitestate machines from inputoutput strings. Parallel work has studied the computational power of DTRNN in connection with finitestate computation. This paper describes a simple strategy to devise stable encodings of finitestate machines in computationally capable discretetime recurrent neural architectures with sigmoid units, and gives a detailed presentation on how this strategy may be applied to encode a general class of finitestate machines in a variety of commonlyused first and secondorder recurrent neural networks. Unlike previous work that either imposed some restrictions to state values, or used a detailed analysis based on fixedpoint attractors, the present approach applies to any positive, bounded, strictly growing, continuous activation function, and uses simple bounding criteri...
Fuzzy Finitestate Automata Can Be Deterministically Encoded into Recurrent Neural Networks
, 1996
"... There has been an increased interest in combining fuzzy systems with neural networks because fuzzy neural systems merge the advantages of both paradigms. On the one hand, parameters in fuzzy systems have clear physical meanings and rulebased and linguistic information can be incorporated into adapt ..."
Abstract

Cited by 13 (5 self)
 Add to MetaCart
There has been an increased interest in combining fuzzy systems with neural networks because fuzzy neural systems merge the advantages of both paradigms. On the one hand, parameters in fuzzy systems have clear physical meanings and rulebased and linguistic information can be incorporated into adaptive fuzzy systems in a systematic way. On the other hand, there exist powerful algorithms for training various neural network models. However, most of the proposed combined architectures are only able to process static inputoutput relationships, i.e. they are not able to process temporal input sequences of arbitrary length. Fuzzy finitestate automata (FFAs) can model dynamical processes whose current state depends on the current input and previous states. Unlike in the case of deterministic finitestate automata (DFAs), FFAs are not in one particular state, rather each state is occupied to some degree defined by a membership function. Based on previous work on encoding DFAs in discretetim...
The Computational Power of Discrete Hopfield Nets with Hidden Units
 Neural Computation
, 1996
"... We prove that polynomial size discrete Hopfield networks with hidden units compute exactly the class of Boolean functions PSPACE/poly, i.e., the same functions as are computed by polynomial spacebounded nonuniform Turing machines. As a corollary to the construction, we observe also that networks wi ..."
Abstract

Cited by 11 (6 self)
 Add to MetaCart
We prove that polynomial size discrete Hopfield networks with hidden units compute exactly the class of Boolean functions PSPACE/poly, i.e., the same functions as are computed by polynomial spacebounded nonuniform Turing machines. As a corollary to the construction, we observe also that networks with polynomially bounded interconnection weights compute exactly the class of functions P/poly, i.e., the class computed by polynomial timebounded nonuniform Turing machines.
An Overview Of The Computational Power Of Recurrent Neural Networks
 Proceedings of the 9th Finnish AI Conference STeP 2000{Millennium of AI, Espoo, Finland (Vol. 3: "AI of Tomorrow": Symposium on Theory, Finnish AI Society
, 2000
"... INTRODUCTION The two main streams of neural networks research consider neural networks either as a powerful family of nonlinear statistical models, to be used in for example pattern recognition applications [6], or as formal models to help develop a computational understanding of the brain [10]. His ..."
Abstract

Cited by 10 (3 self)
 Add to MetaCart
INTRODUCTION The two main streams of neural networks research consider neural networks either as a powerful family of nonlinear statistical models, to be used in for example pattern recognition applications [6], or as formal models to help develop a computational understanding of the brain [10]. Historically, the brain theory interest was primary [32], but with the advances in computer technology, the application potential of the statistical modeling techniques has shifted the balance. 1 The study of neural networks as general computational devices does not strictly follow this division of interests: rather, it provides a general framework outlining the limitations and possibilities aecting both research domains. The prime historic example here is obviously Minsky's and Papert's 1969 study of the computational limitations of singlelayer perceptrons [34], which was a major inuence in turning away interest from neural network learning to symbolic AI techniques for more
On the Correspondence between Neural Folding Architectures and Tree Automata
, 1998
"... The folding architecture together with adequate supervised training algorithms is a special recurrent neural network model designed to solve inductive inference tasks on structured domains. Recently, the generic architecture has been proven as a universal approximator of mappings from rooted labeled ..."
Abstract

Cited by 10 (1 self)
 Add to MetaCart
The folding architecture together with adequate supervised training algorithms is a special recurrent neural network model designed to solve inductive inference tasks on structured domains. Recently, the generic architecture has been proven as a universal approximator of mappings from rooted labeled ordered trees to real vector spaces. In this article we explore formal correspondences to the automata (language) theory in order to characterize the computational power (representational capabilities) of different instances of the generic folding architecture. As the main result we prove that simple instances of the folding architecture have the computational power of at least the class of deterministic bottomup tree automata. It is shown how architectural constraints like the number of layers, the type of the activation functions (firstorder vs. higherorder) and the transfer functions (threshold vs. sigmoid) influence the representational capabilities. All proofs are carried out in a c...