An approach to connectionist natural language processing is proposed, which is based on hierarchically organized modular Parallel Distributed Processing (PDP) networks and a central lexicon of distributed input/output representations. The modules communicate using these representations, which are global and publicly available in the system. The representations are developed automatically by all networks while they are learning their processing tasks. The resulting representations reflect the regularities in the subtasks, which facilitates robust processing in the face of noise and damage, supports improved generalization, and provides expectations about possible contexts. The lexicon can be extended by cloning new instances of the items, that is, by generating a number of items with known processing properties and distinct identities. This technique combinatorially increases the processing power of the system. The recurrent FGREP module, together with a central lexicon, is used as a ba...

In this paper we propose an extension of the Recursive Auto-Associative Memory (RAAM) by Pollack. This extension, the Labeling RAAM (LRAAM), is able to encode labeled graphs with cycles by representing pointers explicitly. A theoretical analysis of the constraints imposed on the weights by the learning task under the hypothesis of perfect learning and linear output units is presented. Cycles and confluent pointers result to be particularly effective in imposing constraints on the weights. Some technical problems encountered in the RAAM, such as the termination problem in the learning and decoding processes, are solved more naturally in the LRAAM framework. The representations developed for the pointers seem to be robust to recurrent decoding along a cycle. Data encoded in a LRAAM can be accessed by pointer as well as by content. The direct access by content can be achieved by transforming the encoder network of the LRAAM in a Bidirectional Associative Memory (BAM). Different access pro...

This paper will give an overview of the various approaches to neurosymbolic integration. Roughly, these can be divided into two strategies: unified strategies aim at attaining neural and symbolic capabilities using neural networks alone, while hybrid strategies combine neural networks with symbolic models such as expert systems, case-based reasoning systems, 2 Chapter 2 and decision trees. These two approaches form the main subtrees of the classification hierarchy depicted in Figure 1. Symbol Proc. Neuronal Unified approach Symbol Proc. hybrids Connectionist Localist Hybrid approach Combined L/D Neurosymbolic integration Functional Chainprocessing Translational Subprocessing hybrids Metaprocessing Distributed Coprocessing Figure 1 Classification of integrated neurosymbolic systems.

The purpose of this book is to present a collection of papers that represents a broad spectrum of current research in learning methods for natural language processing, and to advance the state of the art in language learning and artificial intelligence. The book should bridge a gap between several areas that are usually discussed separately, including connectionist, statistical, and symbolic methods. In order to bring together new and different language learning approaches, we held a workshop at the International Joint Conference on Artificial Intelligence in Montreal in August 1995. Paper contributions were selected and revised after having been reviewed by at least twomembers of the international program committee as well as additional reviewers. This book contains the revised workshop papers and additional papers by members of the program committee. In particular this book focuses on current issues such as: -- How can we apply existing learning methods to language processing? -- What new learning methods are needed for language processing and why? -- What language knowledge should be learned and why?

This paper describes multidimensional neural preference classes and preference Moore machines as a principle for integrating di erent neural and/or symbolic knowledge sources. We relate neural preferences to multidimensional fuzzy set representations. Furthermore, we introduce neural preference Moore machines and relate traditional symbolic transducers with simple recurrent networks by using neural preference Moore machines. Finally, we demonstrate how the concepts of preference classes and preference Moore machines can be used to integrate knowledge from di erent neural and/or symbolic machines. We argue that our new concepts for preference Moore machines contribute a new potential approach towards general principles of neural symbolic integration. 1

The problem of multiple instantiation is the ability to handle different instances of a unique object at the same time. For connectionist models that do not use a working area containing copies of items from a long-term knowledge base, the problem of multiple instantiation is a particularly difficult one. While people are able to deal with multiple instances, their performance when doing so is nonetheless poorer, which is not the case for symbolic models. A cognitive model should reflect competence, as well as its limits. Some connectionist solutions to the problem of multiple instantiation are mentioned in this paper. An new solution which makes use of semi-distributed representations is presented. This model does not separate the long term knowledge base from a working area and has no recourse to copies. This solution limits the process of multiple instantiation in a way that should better reflect human data. Introduction Multiple instantiation involves the simultan...

We present a scheme for translating high-level descriptions of conceptual hierarchies into a neural network representation. The intuitive semantics of a conceptual hierarchy is provided by a Bayesian net, and the neural network implementation provably approximates the behaviour of this net under a stochastic simulation rule.

ions are regularities that best describe the structure of the data. It might be possible to devise a self-organizing process that is sensitive to the internal structure of the training examples. The network would learn the processing task, and at the same time develop a layout of rules, schemas, and other abstract structures that best describe the data. Further input would then be interpreted and represented in terms of this layout (i.e., in terms of the internal structure of the input). Such a capacity would be a major step toward extending subsymbolic AI beyond the limitations of current models. 11 Conclusion Above all, DISCERN serves to show that subsymbolic high-level AI is feasible. DISCERN constitutes a first implementation of the integrated connectionist approach, demonstrating that it is possible to build complete models from independently designed connectionist components. The scale-up properties of the approach seem quite good. Hierarchical modular structure with sequential ...

An implementation of a distributed connectionist network of spiking neuron-like elements is presented. Spiking nodes fire at a precise moment and transmit their activation, with particular strengths and delays, to nodes connected to them. The receiving nodes accumulate potential, but also slowly their potential through decay. When the potential of the node reaches a particular threshold, it emits a spike. Thereafter, the potential is reset to a resting value. As with real neurons, there is a short refractory period during which this node will be completely insensitive to incoming signals, after which its sensitivity will slowly increase. Precise timing properties are used to represent symbols in a distributed manner and to solve the problems of variable binding and multiple instantiation. Several predictions about human short-term memory, predicate processing, complex reasoning, and multiple instantiation arise from this model. This network shows how symbolic processing can be achieved using neurologically and psychologically plausible mechanisms that also have the advantage of generalization and noise tolerance found in connectionist networks.

This paper describes preference classes and preference Moore machines as a basis for integrating different hybrid neural representations. Preference classes are shown to pro- vide a basic link between neural preferences and fuzzy representations at the preference class level. Preference Moore machines provide a link between recurrent neural networks and symbolic transducers at the preference Moore machine level. We demonstrate how the concepts of preference classes and preference Moore machines can be used to interpret neural network representations and to integrate knowledge from hybrid neural represen- tations. One main contribution of this paper is the introduction and analysis of neural preference Moore machines and their link to a fuzzy interpretation. Furthermore, we il- lustrate the interpretation and combination of various neural preference Moore machines with additional real world examples.