In order to solve the "sensitivity-stability" problem --- and its immediate correlate, the problem of sequential learning --- it is crucial to develop connectionist architectures that are simultaneously sensitive to, but not excessively disrupted by, new input. French (1992) suggested that to alleviate a particularly severe form of this disruption, catastrophic forgetting, it was necessary for networks to dynamically separate their internal representations during learning. McClelland, McNaughton, & O'Reilly (1995) went even further. They suggested that nature's way of implementing this obligatory separation was the evolution of two separate areas of the brain, the hippocampus and the neocortex. In keeping with this idea of radical separation, a "pseudo-recurrent" memory model is presented here that partitions a connectionist network into two functionally distinct, but continually interacting areas. One area serves as a final-storage area for representations; the other is an e...

The dual-network memory model is designed to be a neurobiologically plausible manner of avoiding catastrophic interference. We discuss a number of advantages of this model and potential clues that the model has provided in the areas of memory consolidation, category-specific deficits, anterograde and retrograde amnesia. We discuss a surprising result about how this class of models handles episodic ("snap-shot") memory --- namely, that they seem to be able to handle both episodic and abstract memory --- and discuss two other promising areas of research involving these models. 1.

Everyone agrees that real cognition requires much more than static pattern recognition. In particular, it requires the ability to learn sequences of patterns (or actions) But learning sequences really means being able to learn multiple sequences, one after the other, wi thout the most recently learned ones erasing the previously learned ones. But if catastrophic interference is a problem for the sequential learning of individual patterns, the problem is amplified many times over when multiple sequences of patterns have to be learned consecutively, because each new sequence consists of many linked patterns. In this paper we will present a connectionist architecture that would seem to solve the problem of multiple sequence learning using pseudopatterns.

An unusual asymmetry has been observed in natural category formation in infants (Quinn, Eimas, and Rosenkrantz, 1993). Infants who are initially exposed to a series of pictures of cats and then are shown a dog and a novel cat, show significantly more interest in the dog than in the cat. However, when the order of presentation is reversed --- dogs are seen first, then a cat and a novel dog --- the cat attracts no more attention than the dog. We show that a simple connectionist network can model this unexpected learning asymmetry and propose that this asymmetry arises naturally from the asymmetric overlaps of the feature distributions of the two categories. The values of the cat features are subsumed by those of dog features, but not vice-versa. The autoencoder used for the experiments presented in this paper also reproduces exclusivity effects in the two categories as well the reported effect of catastrophic interference of dogs on previously learned cats, but not vice-...

Most artificial neural networks suffer from the problem of catastrophic for-getting, where previously learnt information is suddenly and completely lost when new information is learnt. Memory in real neural systems does not appear to suffer from this unusual behaviour. In this thesis we discuss the problem of catastrophic forgetting in Hopfield networks, and investi-gate various potential solutions. We extend the pseudorehearsal solution of Robins (1995) enabling it to work in this attractor network, and compare the results with the unlearning procedure proposed by Crick and Mitchison (1983). We then explore a familiarity measure based on the energy profile of the learnt patterns. By using the ratio of high energy to low energy parts of the network we can robustly distinguish the learnt patterns from the large number of spurious “fantasy ” patterns that are common in these networks. This energy ratio measure is then used to improve the pseudorehearsal solu-tion so that it can store 0.3N patterns in the Hopfield network, significantly

Category-specific semantic deficits refer to the inability to name objects from a particular category while the naming of words outside that category remains relatively unimpaired. We suggest that such semantic deficits arise from the random lesioning of a unified semantic network in which internal category representations reflect the variability of the categories themselves. This is demonstrated by lesioning networks that have learned to categorize butterflies and chairs. The model shows category-specific semantic deficits of the narrower category (butterfly) with the occasional reverse semantic deficits of the relatively broader category (chair) . Introduction Category-specific semantic deficits refer to the inability to name objects from a particular category as a result of neurological damage. The naming of objects outside the impaired category is relatively well preserved. Perhaps the most striking category-specific semantic deficit is the dissociation found between a...

Abstract. Consolidation has been implemented in two ways: as straight rehearsal of patterns or as pseudorehearsal, in which pseudoitems are created by sampling attractors or input–output combinations from the network. Although both implementations have been investigated by several authors, few have explored how it is decided which pattern or pseudoitem is consolidated. Controlling consolidation is not trivial, as it is susceptible to a corruption. In runaway consolidation, one or two patterns monopolize all consolidation resources and come to dominate the entire network. Runaway consolidation is analysed, and three solutions are explored. Suppressing transmission in the connections in which consolidation takes place is shown to work best. Placing bounds on connections or unlearning attractors also alleviates runaway consolidation, though less effectively so.

A key claim of current theoretical analyses of the memory impairments associated with amnesia is that certain distinct forms of learning and memory are spared. A compelling example is that amnesic patients and controls are indistinguishable in repetition priming but amnesic patients are impaired at recognizing the study items. The authors show that this pattern of results is predicted by a single-system connectionist model of learning in which amnesia is simulated by a reduced learning rate. They also demonstrate that the model can reproduce the converse pattern in which priming but not recognition is impaired if the input is assumed to be additionally degraded in a priming test. The authors conclude that dissociations between priming and recognition do not require functionally or neurally distinct memory systems. According to an influential view, memory is not a unitary faculty but is composed of multiple systems that work independently of each other (Gabrieli, 1998; Squire, 1994). The most prominent distinction that has been proposed is between declarative and nondeclarative memory. Declarative (or explicit) memory is usually characterized by the conscious and intentional recollection of knowledge. Typical tests of declarative memory involve

An hypothesis of functional transfer of task knowledge is presented that requires the development of a measure of task relatedness and a method of sequential learning. The task rehearsal method (TRM) is introduced to address the issues of sequential learning, namely retention and transfer of knowledge. TRM is a knowledge based inductive learning system that uses functional domain knowledge as a source of inductive bias. The representations of successfully learned tasks are stored within domain knowledge. Virtual examples generated by domain knowledge are rehearsed in parallel with the each new task using either the standard multiple task learning (MTL) or the jMTL neural network methods. The results of experiments conducted on a synthetic domain of seven tasks demonstrate the method's ability to retain and transfer task knowledge. TRM is shown to be effective in developing hypothesis for tasks that suffer from impoverished training sets. Difficulties encountered during sequential learn...

: : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : x 1 Introduction : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : 1 1.1 Thesis Outline : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : 3 2 Speech Recognition with Neural Networks : : : : : : : : : : : : : : : : : : 4 2.1 The Speech Recognition Problem : : : : : : : : : : : : : : : : : : : : : : : : 4 2.2 Hybrid Systems : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : 7 2.2.1 Architecture : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : 8 2.2.2 Evaluation and Training : : : : : : : : : : : : : : : : : : : : : : : : : 8 3 Review of Adaptation Literature : : : : : : : : : : : : : : : : : : : : : : : : 13 3.1 Speaker Adaptation/Normalization : : : : : : : : : : : : : : : : : : : : : : : 13 3.1.1 Speaker Categorization Approaches : : : : : : : : : : : : : : : : : : : 16 3.1.2 Data/Feature Transformation Approaches : : : : : : : : : ...