Results 1  10
of
39
On The Computational Power Of Neural Nets
 JOURNAL OF COMPUTER AND SYSTEM SCIENCES
, 1995
"... This paper deals with finite size networks which consist of interconnections of synchronously evolving processors. Each processor updates its state by applying a "sigmoidal" function to a linear combination of the previous states of all units. We prove that one may simulate all Turing Machines by su ..."
Abstract

Cited by 156 (26 self)
 Add to MetaCart
This paper deals with finite size networks which consist of interconnections of synchronously evolving processors. Each processor updates its state by applying a "sigmoidal" function to a linear combination of the previous states of all units. We prove that one may simulate all Turing Machines by such nets. In particular, one can simulate any multistack Turing Machine in real time, and there is a net made up of 886 processors which computes a universal partialrecursive function. Products (high order nets) are not required, contrary to what had been stated in the literature. Nondeterministic Turing Machines can be simulated by nondeterministic rational nets, also in real time. The simulation result has many consequences regarding the decidability, or more generally the complexity, of questions about recursive nets.
A Survey of Computational Complexity Results in Systems and Control
, 2000
"... The purpose of this paper is twofold: (a) to provide a tutorial introduction to some key concepts from the theory of computational complexity, highlighting their relevance to systems and control theory, and (b) to survey the relatively recent research activity lying at the interface between these fi ..."
Abstract

Cited by 116 (21 self)
 Add to MetaCart
The purpose of this paper is twofold: (a) to provide a tutorial introduction to some key concepts from the theory of computational complexity, highlighting their relevance to systems and control theory, and (b) to survey the relatively recent research activity lying at the interface between these fields. We begin with a brief introduction to models of computation, the concepts of undecidability, polynomial time algorithms, NPcompleteness, and the implications of intractability results. We then survey a number of problems that arise in systems and control theory, some of them classical, some of them related to current research. We discuss them from the point of view of computational complexity and also point out many open problems. In particular, we consider problems related to stability or stabilizability of linear systems with parametric uncertainty, robust control, timevarying linear systems, nonlinear and hybrid systems, and stochastic optimal control.
Turing Computability With Neural Nets
 Applied Mathematics Letters
, 1991
"... . This paper shows the existence of a finite neural network, made up of sigmoidal neurons, which simulates a universal Turing machine. It is composed of less than 10 5 synchronously evolving processors, interconnected linearly. Highorder connections are not required. 1. Introduction This paper a ..."
Abstract

Cited by 60 (13 self)
 Add to MetaCart
. This paper shows the existence of a finite neural network, made up of sigmoidal neurons, which simulates a universal Turing machine. It is composed of less than 10 5 synchronously evolving processors, interconnected linearly. Highorder connections are not required. 1. Introduction This paper addresses the question: What ultimate limitations, if any, are imposed by the use of neural nets as computing devices? In particular, and ignoring issues of training and practicality of implementation, one would like to know if every problem that can be solved by a digital computer is also solvable in principle using a net. This question has been asked before in the literature. Indeed, Jordan Pollack ([7]) showed that a certain recurrent net model which he called a "neuring machine," for "neural Turing" is universal. In his model, all neurons synchronously update their states according to a quadratic combination of past activation values. In general, one calls highorder nets those in...
The dynamic universality of sigmoidal neural networks
 Inf. Comput
, 1996
"... We investigate the computational power of recurrent neural networks that apply the sigmoid activation function _(x)=[2 (1+e &x)]&1. These networks are extensively used in automatic learning of nonlinear dynamical behavior. We show that in the noiseless model, there exists a universal architecture t ..."
Abstract

Cited by 35 (2 self)
 Add to MetaCart
We investigate the computational power of recurrent neural networks that apply the sigmoid activation function _(x)=[2 (1+e &x)]&1. These networks are extensively used in automatic learning of nonlinear dynamical behavior. We show that in the noiseless model, there exists a universal architecture that can be used to compute any recursive (Turing) function. This is the first result of its kind for the sigmoid activation function; previous techniques only applied to linearized and truncated version of this function. The significance of our result, besides the proving technique itself, lies in the popularity of the sigmoidal function both in engineering applications of artificial neural networks and in biological modelling. Our techniques can be applied to a much more general class of ``sigmoidallike' ' activation functions, suggesting that Turing universality is a relatively common property of recurrent neural network models.] 1996 Academic Press, Inc. 1.
Recurrent Networks: State Machines Or Iterated Function Systems?
 Proceedings of the 1993 Connectionist Models Summer School
, 1994
"... this paper, clustering of hidden unit activations, or recurrent network state space, provides incomplete information regarding the IP state of the network. IP states determine future behavior as well as encapsulate input history. The network's state transformations can exhibit sensitivity to initial ..."
Abstract

Cited by 25 (1 self)
 Add to MetaCart
this paper, clustering of hidden unit activations, or recurrent network state space, provides incomplete information regarding the IP state of the network. IP states determine future behavior as well as encapsulate input history. The network's state transformations can exhibit sensitivity to initial conditions and generate disparate futures for state clusters of all sizes. The second part of the paper presents IFS theory and shows how it can explain recurrent network state dynamics. By linking IFSs and recurrent networks, existing constraints on network dynamics independent of network models are now evident. By assuming a finite set of inputs, which is often the case in symbolic domains, one can describe recurrent network models as a finite collection of nonlinear state transformations.The interaction of several transforms produces complex state spaces with recursive structure. The limit behavior of the collection of transformations, and recurrent networks in symbolic applications, is more complex than the union of the individual transformations. An input driven recurrent network behaves like the random iteration algorithm. Infinite input sequence generates sequences of points dense in the state space attractor when the transformations are contractive. While the demonstration in this paper used the SCN, other models produce similar IFSlike behaviors as long as the network's input selects transformations [19]. The IFS approach also explains the phenomena of state clustering in recurrent networks. In [20], ServenSchreiber et al report significant clustering in simple recurrent networks [21] both before and after training from the Reber grammar prediction task. A set of random transformations will normally reduce the volume of the recurrent networks state space, and plac...
Computational Complexity Of Neural Networks: A Survey
, 1994
"... . We survey some of the central results in the complexity theory of discrete neural networks, with pointers to the literature. Our main emphasis is on the computational power of various acyclic and cyclic network models, but we also discuss briefly the complexity aspects of synthesizing networks fr ..."
Abstract

Cited by 22 (6 self)
 Add to MetaCart
. We survey some of the central results in the complexity theory of discrete neural networks, with pointers to the literature. Our main emphasis is on the computational power of various acyclic and cyclic network models, but we also discuss briefly the complexity aspects of synthesizing networks from examples of their behavior. CR Classification: F.1.1 [Computation by Abstract Devices]: Models of Computationneural networks, circuits; F.1.3 [Computation by Abstract Devices ]: Complexity Classescomplexity hierarchies Key words: Neural networks, computational complexity, threshold circuits, associative memory 1. Introduction The currently again very active field of computation by "neural" networks has opened up a wealth of fascinating research topics in the computational complexity analysis of the models considered. While much of the general appeal of the field stems not so much from new computational possibilities, but from the possibility of "learning", or synthesizing networks...
Connectionist, Statistical and Symbolic Approaches to Learning for Natural Language Processing
, 1996
"... The purpose of this book is to present a collection of papers that represents a broad spectrum of current research in learning methods for natural language processing, and to advance the state of the art in language learning and artificial intelligence. The book should bridge a gap between several a ..."
Abstract

Cited by 18 (10 self)
 Add to MetaCart
The purpose of this book is to present a collection of papers that represents a broad spectrum of current research in learning methods for natural language processing, and to advance the state of the art in language learning and artificial intelligence. The book should bridge a gap between several areas that are usually discussed separately, including connectionist, statistical, and symbolic methods. In order to bring together new and different language learning approaches, we held a workshop at the International Joint Conference on Artificial Intelligence in Montreal in August 1995. Paper contributions were selected and revised after having been reviewed by at least twomembers of the international program committee as well as additional reviewers. This book contains the revised workshop papers and additional papers by members of the program committee. In particular this book focuses on current issues such as:  How can we apply existing learning methods to language processing?  What new learning methods are needed for language processing and why?  What language knowledge should be learned and why?
The Neural Network Pushdown Automaton: Model, Stack and Learning Simulations
, 1993
"... In order for neural networks to learn complex languages or grammars, they must have sufficient computational power or resources to recognize or generate such languages. Though many approaches to effectively utilizing the computational power of neural networks have been discussed, an obvious one is t ..."
Abstract

Cited by 17 (2 self)
 Add to MetaCart
In order for neural networks to learn complex languages or grammars, they must have sufficient computational power or resources to recognize or generate such languages. Though many approaches to effectively utilizing the computational power of neural networks have been discussed, an obvious one is to couple a recurrent neural network with an external stack memory in effect creating a neural network pushdown automata (NNPDA). This NNPDA generalizes the concept of a recurrent network so that the network becomes a more complex computing structure. This paper discusses in detail a NNPDA its construction, how it can be trained and how useful symbolic information can be extracted from the trained network. To effectively couple the external stack to the neural network, an optimization method is developed which uses an error function that connects the learning of the state automaton of the neural network to the learning of the operation of the external stack: push, pop, and nooperation. To minimize the error function using gradient descent learning, an analog stack is designed such that the action and storage of information in the stack are continuous. One interpretation of a continuous stack is the probabilistic storage of and action on data. After training on sample strings of an unknown source grammar, a quantization procedure extracts from the analog stack and neural network a discrete pushdown automata (PDA). Simulations show that in learning deterministic contextfree grammars the balanced parenthesis language, 1 n 0 n, and the deterministic Palindrome the extracted PDA is correct in the sense that it can correctly recognize unseen strings of arbitrary length. In addition, the extracted PDAs can be shown to be identical or equivalent to the PDAs of the source grammars which were used to generate the training strings.
A Competitive Attachment Model for Resolving Syntactic Ambiguities in Natural Language Parsing
, 1994
"... Linguistic ambiguity is the greatest obstacle to achieving practical computational systems for natural language understanding. By contrast, people experience surprisingly little difficulty in interpreting ambiguous linguistic input. This dissertation explores distributed computational techniques for ..."
Abstract

Cited by 14 (4 self)
 Add to MetaCart
Linguistic ambiguity is the greatest obstacle to achieving practical computational systems for natural language understanding. By contrast, people experience surprisingly little difficulty in interpreting ambiguous linguistic input. This dissertation explores distributed computational techniques for mimicking the human ability to resolve syntactic ambiguities efficiently and effectively. The competitive attachment theory of parsing formulates the processing of an ambiguity as a competition for activation within a hybrid connectionist network. Determining the grammaticality of an input relies on a new approach to distributed communication that integrates numeric and symbolic constraints on passing features through the parsing network. The method establishes syntactic relations both incrementally and efficiently, and underlies the ability of the model to establish longdistance syntactic relations using only local communication within a network. The competitive distribution of numeric ev...
Neural Networks and Complexity Theory
 In Proc. 17th International Symposium on Mathematical Foundations of Computer Science
, 1992
"... . We survey some of the central results in the complexity theory of discrete neural networks, with pointers to the literature. 1 Introduction The recently revived field of computation by "neural" networks provides the complexity theorist with a wealth of fascinating research topics. While much of ..."
Abstract

Cited by 14 (4 self)
 Add to MetaCart
. We survey some of the central results in the complexity theory of discrete neural networks, with pointers to the literature. 1 Introduction The recently revived field of computation by "neural" networks provides the complexity theorist with a wealth of fascinating research topics. While much of the general appeal of the field stems not so much from new computational possibilities, but from the possibility of "learning", or synthesizing networks directly from examples of their desired inputoutput behavior, it is nevertheless important to pay attention also to the complexity issues: firstly, what kinds of functions are computable by networks of a given type and size, and secondly, what is the complexity of the synthesis problems considered. In fact, inattention to these issues was a significant factor in the demise of the first stage of neural networks research in the late 60's, under the criticism of Minsky and Papert [51]. The intent of this paper is to survey some of the centra...