## Analysis of Dynamical Recognizers (1996)

### Cached

### Download Links

- [www.csse.unimelb.edu.au]
- [www.demo.cs.brandeis.edu]
- [www.cse.unsw.edu.au]
- [demo.cs.brandeis.edu]
- [www.cse.unsw.edu.au]
- [www.csee.uq.edu.au]
- DBLP

### Other Repositories/Bibliography

Venue: | NEURAL COMPUTATION |

Citations: | 33 - 5 self |

### BibTeX

@ARTICLE{Blair96analysisof,

author = {Alan D. Blair and Jordan B. Pollack},

title = {Analysis of Dynamical Recognizers},

journal = {NEURAL COMPUTATION},

year = {1996},

volume = {9},

pages = {1127--1142}

}

### OpenURL

### Abstract

Pollack (1991) demonstrated that second-order recurrent neural networks can act as dynamical recognizers for formal languages when trained on positive and negative examples, and observed both phase transitions in learning and IFS-like fractal state sets. Follow-on work focused mainly on the extraction and minimization of a finite state automaton (FSA) from the trained network. However, such networks are capable of inducing languages which are not regular, and therefore not equivalenttoany FSA. Indeed, it may be simpler for a small network to fit its training data by inducing such a non-regular language. But when is the network's language not regular? In this paper, using a low dimensional network capable of learning all the Tomita data sets, we present an empirical method for testing whether the language induced by the network is regular or not. We also provide a detailed "-machine analysis of trained networks for both regular and non-regular languages.

### Citations

781 |
M.,"Fractals Everywhere
- Barnsley
- 1993
(Show Context)
Citation Context ...phenomenon of regular vs. non-regular network behavior in general. Finally,we remark that the functions w 0 and w 1 map X continuously into itself and as such de#ne an IteratedFunction System or IFS #=-=Barnsley, 1988-=-# as noted in #Kolen, 1994#. The attractor A of this IFS may be de#ned as follows: #i# Z 0 = X #ii# For i # 0, Z i+1 = w 0 #Z i # # w 1 #Z i # #by induction Z i+1 #Z i # #iii# A = T i#0 Z i As with A ... |

438 | A Learning Algorithm for Continually Running Fully Recurrent Neural Networks - Williams, Zipser - 1980 |

304 | Introduction to Automata Theory - Hopcroft, Motwani, et al. - 2001 |

301 | Learning Representations by BackPropagating Errors. Nature 323 - Rumelhart, Hinton, et al. - 1986 |

269 |
Faster-Learning Variations on BackPropagation: An Empirical Study
- Fahlman
- 1988
(Show Context)
Citation Context ...e architecture described in x2were trained to recognize formal languages using backpropagation through time #Williams & Zipser, 1989, Rumelhart et al., 1986#, with a modi#cation similar to Quickprop #=-=Fahlman, 1989-=-# 1 and a learning rate of 0:03. The weights were updated in batch mode, and the perceptron weights P j 1 Speci#cally, the cost function we used was E = # 1 2 #1 + s# 2 log# 1+ z 1+ s # # 1 2 #1 # s# ... |

267 |
Attractor dynamics and parallelism in a connectionist sequential machine
- Jordan
- 1986
(Show Context)
Citation Context ...anguages from examples. The resulting networks often displayed complex limit dynamics whichwere fractal in nature #Kolen, 1993#. Alternative architectures had been employed earlier for related tasks #=-=Jordan, 1986-=-, Pollack, 1987, Cleeremans et al., 1989#. Others have been proposed since #Watrous & Kuhn, 1992, Frasconi et al., 1992, Zeng et al., 1994, Das & Mozer, 1994, Forcada & Carrasco, 1995# and a number of... |

210 | The induction of dynamical recognizers
- Pollack
- 1991
(Show Context)
Citation Context ... t = w # t #x t#1 # given by x j t = tanh #W j0 # t + d X k=1 W jk # t x k t#1 #; for 1 # j # d; 1 # t # n: This part of the architecture is equivalent to the second order recurrent networks used in #=-=Pollack, 1991-=-# and #Giles et al., 1992#, with a slightly di#erent notation. After the whole string has been read, the #nal state x n is fed through a perceptron P producing output z = tanh #P 0 + d X j=1 P j x j n... |

176 |
Learning and extracting finite state automata with second-order recurrent neural networks
- Giles, Miller, et al.
- 1992
(Show Context)
Citation Context ...etworks have been developed. One of the principal themes has been the use of clustering and minimization techniques to extract a Finite State Automaton #FSA# that approximates the network's behavior #=-=Giles et al., 1992-=-, Manolios & Fanelli, 1994, Ti#no & Sajda, 1995#. Casey #1996# showed that if the network robustly models an FSA, the method proposed in #Giles et al., 1992# will successfully extract the FSA given a ... |

154 | Finite state automata and simple recurrent networks - Cleeremans, Servan-Schreiber, et al. - 1989 |

101 |
The dynamics of discrete-time computation, with application to recurrent neural networks and finite state machine extraction”, Neural Computation 8
- Casey
- 1135
(Show Context)
Citation Context ...ts from #Tomita, 1982# within a few hundred training epochs. 3 Analysis Our purpose here is not only to train the network but also to gain some insightinto how it accomplishes its assigned task #see #=-=Casey, 1996-=-# for another, related approach to this problem#. If the recurrentlayer has d nodes, the state space of the system is the d-dimensional hypercube X =##1;1# d .However we need not be concerned with the... |

91 | Computation at the onset of chaos - Crutchfield, Young - 1990 |

70 | Random DFAs can be approximately learned from sparse uniform sample
- Lang
- 1992
(Show Context)
Citation Context ...qually valid induced language that may be regular or non-regular. Good symbolic algorithms exist already for #nding a regular language compatible with given input data #Trakhtenbrot & Barzdin', 1973, =-=Lang, 1992-=-#. Our purpose is rather to analyse the kinds of languages that a dynamical system such as a neural network is likely to come up with when trained on that data. We do not claim that our methods are e#... |

54 | On the computational power of neural networks - Siegelmann, Sontag - 1995 |

48 |
1994] “The calculi of emergence
- Crutchfield
(Show Context)
Citation Context ...ximation A 0 for A 0 . In discrete form, the above procedure may be equated with the Hopcroft Minimization Algorithm (Hopcroft & Ullman, 1979), or the method of "-machines (Crutchfield & Young, 1=-=990, Crutchfield, 1994-=-), and was first used in the present context by Giles et al. (1992). Using small values of r, their aim was to extract an FSA that, while it might not model the network's behavior exactly, would model... |

39 |
Dynamic construction of finite-state automata from examples using hill-climbing
- Tomita
- 1982
(Show Context)
Citation Context ... of 17 free parameters -- 3 for the perceptron, 6 for each subnetwork and 2 for the initial point. As outlined below, this architecture was adequate to the task of learning any of the data sets from (=-=Tomita, 1982-=-) within a few hundred training epochs. 3 Analysis Our purpose here is not only to train the network but also to gain some insight into how it accomplishes its assigned task [see (Casey, 1996) for ano... |

32 | A unified gradient-descent/clustering architecture for finite state machine induction - Das, Mozer - 1994 |

31 | First order recurrent neural networks and deterministic finite state automata - Manolios, Fanelli - 1994 |

28 |
Cascaded back-propagation on dynamic connectionist networks
- Pollack
- 1987
(Show Context)
Citation Context ...examples. The resulting networks often displayed complex limit dynamics whichwere fractal in nature #Kolen, 1993#. Alternative architectures had been employed earlier for related tasks #Jordan, 1986, =-=Pollack, 1987-=-, Cleeremans et al., 1989#. Others have been proposed since #Watrous & Kuhn, 1992, Frasconi et al., 1992, Zeng et al., 1994, Das & Mozer, 1994, Forcada & Carrasco, 1995# and a number of approaches to ... |

25 | Fool’s gold: Extracting finite state machines from recurrent network dynamics
- Kolen
- 1994
(Show Context)
Citation Context ...ollack #1991# showed one way a recurrent network may be trained to recognize formal languages from examples. The resulting networks often displayed complex limit dynamics whichwere fractal in nature #=-=Kolen, 1993-=-#. Alternative architectures had been employed earlier for related tasks #Jordan, 1986, Pollack, 1987, Cleeremans et al., 1989#. Others have been proposed since #Watrous & Kuhn, 1992, Frasconi et al.,... |

21 | Learning the initial state of a Second-Order Recurrent Neural Network during regular-language inference - Forcada, Carrasco - 1995 |

21 |
Exploring the Computational Capabilities of Recurrent Neural Networks
- Kolen
- 1994
(Show Context)
Citation Context ...-regular network behavior in general. Finally,we remark that the functions w 0 and w 1 map X continuously into itself and as such de#ne an IteratedFunction System or IFS #Barnsley, 1988# as noted in #=-=Kolen, 1994-=-#. The attractor A of this IFS may be de#ned as follows: #i# Z 0 = X #ii# For i # 0, Z i+1 = w 0 #Z i # # w 1 #Z i # #by induction Z i+1 #Z i # #iii# A = T i#0 Z i As with A 0 ,we can #nd a discrete a... |

20 | Induction of finite state languages using second-order recurrent networks - Watrous, Kuhn - 1992 |

15 | Learning and extracting initial Mealy automata with a modular neural network model - Tino, Sajda - 1995 |

14 | Recurrent neural networks and prior knowledge for sequence processing: a constrained non-deterministic approach - Frasconi, Gori, et al. - 1995 |

5 | Extraction of Rules from - Omlin, Giles - 1996 |

3 |
Dynamic construction of �nite-state automata from examples using hill-climbing
- Tomita
- 1982
(Show Context)
Citation Context ...l of 17 free parameters # 3 for the perceptron, 6 for each subnetwork and 2 for the initial point. As outlined below, this architecture was adequate to the task of learning any of the data sets from #=-=Tomita, 1982-=-# within a few hundred training epochs. 3 Analysis Our purpose here is not only to train the network but also to gain some insightinto how it accomplishes its assigned task #see #Casey, 1996# for anot... |

2 | Computation Dynamics in Discrete-Time Recurrent Neural Networks - Casey - 1993 |

2 | Computation at the onset of chaos - Crutch�eld, Young - 1990 |

1 | The Calculi of Emergence - Crutcheld - 1994 |