Abstract

A higher order recurrent neural network architecture learns to recognize and generate languages after being "trained " on categorized exemplars. Studying these networks from the perspective of dynamical systems yields two interesting discoveries: First, a longitudinal examination of the learning process illustrates a new form of mechanical inference: Induction by phase transition. A small weight adjustment causes a "bifurcation" in the limit behavior of the network. This phase transition corresponds to the onset of the network’s capacity for generalizing to arbitrary-length strings. Second, a study of the automata resulting from the acquisition of previously published training sets indicates that while the architecture is not guaranteed to find a minimal finite automaton consistent with the given exemplars, which is an NP-Hard problem, the architecture does appear capable of generating non-regular languages by exploiting fractal and chaotic dynamics. I end the paper with a hypothesis relating linguistic generative capacity to the behavioral regimes of non-linear dynamical systems.

Citations

1313	Finding structure in time - Elman - 1990
1297	Neural networks and physical systems with emergent collective computational abilities - Hopfield - 1982
1159	Fractal Geometry of Nature - Mandelbrot - 1982
1027	Aspects of the Theory of Syntax - Chomsky - 1965
976	Multilayer feedforward networks are universal approximators - Hornik, Stinchcombe, et al. - 1989
549	A logical calculus of the ideas immanent in nervous activity - McCulloch, Pitts - 1943
545	Fractals Everywhere - Barnsley - 1988
488	Connectionism and cognitive architecture: a critical analysis - Fodor, Pylyshyn - 1988
352	Computation: Finite and Infinite Machines - Minsky - 1967
299	Recursive distributed representations - Pollack - 1990
286	Inductive inference: Theory and methods - ANGLUIN, SMITH - 1983
261	An introduction to chaotic dynamical systems - Devaney - 1986
255	Three models for the description of language - Chomsky - 1956
248	Language Learnability and Language Development - Pinker - 1984
217	On language and connectionism: Analysis of a parallel distributed processing model of language acquisition - Pinker - 1988
210	Chaos: Making a New Science - Gleick - 1987
199	Measuring the strangeness of a strange attractor - Grassberger, Procaccia - 1983
195	Complexity of automaton identification from given data - Gold - 1978
191	Information processing in dynamical systems: Foundations of harmony theory - Smolensky - 1986
186	On the length of programs for computing finite binary sequences - Chaitin - 1966
150	Learning state space trajectories in recurrent neural networks - Pearlmutter - 1989
148	Formal Principles of Language Acquisition - Wexler, Culicover - 1980
131	The acquisition of syntactic knowledge - Berwick - 1985
126	Perceptrons - Minsky, Papert - 1969
124	Generalization of backpropagation to recurrent neural networks - Pineda - 1987
122	The convergence of mildly context-sensitive grammatical formalisms - Joshi, Vijay-Shanker, et al. - 1991
112	Tree Adjoining Grammars: How much context sensitivity is required to provide a reasonable structural description - Joshi - 1985
100	Distinctive features, categorical perception, and probability learning: Some applications of a neural model - Anderson, Silverstein, et al. - 1977
88	On the complexity of minimum inference of regular sets - Angluin - 1978
80	A method for synthesizing sequential circuits - Mealy - 1955
79	Back propagation is sensitive to initial conditions - Kolen, Pollack - 1991
77	Computation at the onset of chaos - Crutchfield, Young - 1990
77	Unpredictability and undecidability in dynamical systems - Moore - 1990
66	A focused back-propagation algorithm for temporal pattern recognition - Mozer - 1989
62	The Biology and Evolution of Language - Lieberman - 1984
54	Encoding Sequential Structure in Simple Recurrent Networks - Servan-Schreiber, Cleeremans, et al. - 1989
44	Universal Computation in Simple One-Dimensional Cellular Automata - Lindgren, Nordahl - 1990
39	Higher order recurrent networks and grammatical inference - Giles, Sun, et al. - 1990
36	Dynamic construction of finite-state automata from examples using hill-climbing - Tomita - 1982
29	A new approach to unsupervised learning in deterministic environments - Rivest - 1987
29	PDP Models and general issues in cognitive science - Rumelhart, McClelland - 1986
23	Cascaded Back Propagation on Dynamic Connectionist Networks - Pollack - 1987
19	Intuition in insight and noninsight problem solving. Mem Cognit 15 - Metcalfe, Wiebe - 1987
13	Generalized Context-Free Grammars, Head Grammars, and Natural Language - Pollard - 1984
12	A distributed connectionist representation for concept structures - Touretzky, Geva - 1987
11	An Introduction to Computing with Neural Networks - Lippman
8	Learning Internal Representations through Error Propagation - Rumelhart, Hinton, et al. - 1986
7	Pattern Association in a Back-propagation Network: Implications for Child Language Acquisition - Plunkett, Marchman - 1989
4	Chaotic Behavior of a Layered Neural Network - Derrida, Meir - 1988
4	Mechanisms of Language Acquisition - MacWhinney - 1987