Results 1  10
of
47
Connectionist Learning Procedures
 ARTIFICIAL INTELLIGENCE
, 1989
"... A major goal of research on networks of neuronlike processing units is to discover efficient learning procedures that allow these networks to construct complex internal representations of their environment. The learning procedures must be capable of modifying the connection strengths in such a way ..."
Abstract

Cited by 408 (8 self)
 Add to MetaCart
A major goal of research on networks of neuronlike processing units is to discover efficient learning procedures that allow these networks to construct complex internal representations of their environment. The learning procedures must be capable of modifying the connection strengths in such a way that internal units which are not part of the input or output come to represent important features of the task domain. Several interesting gradientdescent procedures have recently been discovered. Each connection computes the derivative, with respect to the connection strength, of a global measure of the error in the performance of the network. The strength is then adjusted in the direction that decreases the error. These relatively simple, gradientdescent learning procedures work well for small tasks and the new challenge is to find ways of improving their convergence rate and their generalization abilities so that they can be applied to larger, more realistic tasks.
Contrastive hebbian learning in the continuous hopfield model
 In
, 1990
"... This pape.r shows that contrastive Hebbian, the algorithm used in mean field learning, can be applied to any continuous Hopfield model. This implies that nonlogistic activation functions as well as self connections are allowed. Contrary to previous approaches, the learning algorithm is derived with ..."
Abstract

Cited by 47 (1 self)
 Add to MetaCart
This pape.r shows that contrastive Hebbian, the algorithm used in mean field learning, can be applied to any continuous Hopfield model. This implies that nonlogistic activation functions as well as self connections are allowed. Contrary to previous approaches, the learning algorithm is derived without considering it a mean field approximation to Boltzmann machine learning. The paper includes a discussion of the conditions under which the function that contrastive Hebbian mini~ mizes can be considered a proper error function, and an analysis of five different training regimes. An appendix provides complete demonstrations and specific instructions on how to implement contrastive Hebbian learning in interactive activation and competition models (a convenient version of the continuous Hopfield model). 1
Learning continuous probability distributions with symmetric diffusion networks
 Cognitive Science
, 1993
"... in this article we present symmetric diffusion networks, a family of networks that instantiate the principles of continuous, stochastic, adaptive and interactive propagation of information. Using methods of Markovlon diffusion theory, we formalize the activation dynamics of these networks and then ..."
Abstract

Cited by 39 (10 self)
 Add to MetaCart
in this article we present symmetric diffusion networks, a family of networks that instantiate the principles of continuous, stochastic, adaptive and interactive propagation of information. Using methods of Markovlon diffusion theory, we formalize the activation dynamics of these networks and then show that they can be trained to reproduce entire muitivariote probability distributions an their outputs using the contrastive Hebbian learning rule (CHL).,We show that CHL performs gradient descent on an error function that captures differences between desired and obtolned continuous multivoriate probability distributions. This allows the learning algorithm to go beyond expected values of output units and to approximate complete probability distributions on continuous muitivariote activation spaces. We argue that learning continuous distributions is an important task underlying a variety of reallife situations that were beyond the scope of previous connectionist networks. Deterministic networks, like back propagation, cannot ieorn this task because they ore limited to learning average values of independent output units. Previous stochastic connectionist networks could learn probobility distributions but they were limited to discrete variables. Simulations show that symmetric diffusion networks can be trained with the CHL rule to opproximate discrete and continuous probability distributions of various types. 1.
Learning continuous attractors in a recurrent net
 In
, 1998
"... seung~belllabs.com One approach to invariant object recognition employs a recurrent neural network as an associative memory. In the standard depiction of the network's state space, memories of objects are stored as attractive fixed points of the dynamics. I argue for a modification of this pi ..."
Abstract

Cited by 33 (6 self)
 Add to MetaCart
(Show Context)
seung~belllabs.com One approach to invariant object recognition employs a recurrent neural network as an associative memory. In the standard depiction of the network's state space, memories of objects are stored as attractive fixed points of the dynamics. I argue for a modification of this picture: if an object has a continuous family of instantiations, it should be represented by a continuous attractor. This idea is illustrated with a network that learns to complete patterns. To perform the task of filling in missing information, the network develops a continuous attractor that models the manifold from which the patterns are drawn. From a statistical viewpoint, the pattern completion task allows a formulation of unsupervised learning in terms of regression rather than density estimation. A classic approach to invariant object recognition is to use a recurrent neural network as an associative memory[l]. In spite of the intuitive appeal and biological
GAL: Networks that grow when they learn and shrink when they forget
 INTERNATIONAL JOURNAL OF PATTERN RECOGNITION AND ARTIFICIAL INTELLIGENCE
, 1991
"... Learning when limited to modification of some parameters has a limited scope; the capability to modify the system structure is also needed to get a wider range of the learnable. In the case of artificial neural networks, learning by iterative adjustment of synaptic weights can only succeed if t ..."
Abstract

Cited by 29 (5 self)
 Add to MetaCart
Learning when limited to modification of some parameters has a limited scope; the capability to modify the system structure is also needed to get a wider range of the learnable. In the case of artificial neural networks, learning by iterative adjustment of synaptic weights can only succeed if the network designer predefines an appropriate network structure, i.e., number of hidden layers, units, and the size and shape of their receptive and projective fields. This paper advocates the view that the network structure should not, as usually done, be determined by trialanderror but should be computed by the learning algorithm. Incremental learning algorithms can modify the network structure by addition and/or removal of units and/or links. A survey of current connectionist literature is given on this line of thought. "Grow and Learn" (GAL) is a new algorithm that learns an association at oneshot due to being incremental and using a local representation. During the socalled...
Memory Maintenance Via Neuronal Regulation
 Neural Computation
, 1998
"... Since their conception half a century ago Hebbian cell assemblies have become a basic term in the Neurosciences, and the idea that learning takes place through synaptic modifications has been accepted as a fundamental paradigm. As synapses undergo continuous metabolic turnover, adopting the stance t ..."
Abstract

Cited by 24 (6 self)
 Add to MetaCart
(Show Context)
Since their conception half a century ago Hebbian cell assemblies have become a basic term in the Neurosciences, and the idea that learning takes place through synaptic modifications has been accepted as a fundamental paradigm. As synapses undergo continuous metabolic turnover, adopting the stance that memories are engraved in the synaptic matrix raises a fundamental problem: How can memories be maintained for very long time periods? We present a novel solution to this longstanding question, that is based on biological evidence of neuronal regulation mechanisms that act to maintain neuronal activity. Our mechanism is developed within the framework of a neural model of associative memory. It is operative in conjunction with random activation of the memory system, and is able to counterbalance degradation of synaptic weights, and to normalize the basins of attraction of all memories. Over long time periods, when the variance of the degradation process becomes important, the memory syste...
Learning symmetry groups with hidden units: Beyond the perceptron
 Physica
, 1986
"... Learning to recognize mirror, rotational and translational symmetries is a difficult problem for massivelyparallel network models. These symmetries cannot be learned by firstorder perceptrons or Hopfield networks, which have no means for incorporating additional adaptive units that are hidden from ..."
Abstract

Cited by 21 (5 self)
 Add to MetaCart
Learning to recognize mirror, rotational and translational symmetries is a difficult problem for massivelyparallel network models. These symmetries cannot be learned by firstorder perceptrons or Hopfield networks, which have no means for incorporating additional adaptive units that are hidden from the input and output layers. We demonstrate that the Boltzmann learning algorithm is capable of finding sets of weights which turn hidden units into useful higherorder feature detectors capable of solving symmetry problems. 1.
Revenge of the 'neurds': Characterizing creative thought in terms of the structure and dynamics of memory
 Creativity Research Journal
"... ABSTRACT: There is cognitive, neurological, and computational support for the hypothesis that defocusing attention results in divergent or associative thought, conducive to insight and finding unusual connections, while focusing attention results in convergent or analytic thought, conducive to rule ..."
Abstract

Cited by 21 (11 self)
 Add to MetaCart
(Show Context)
ABSTRACT: There is cognitive, neurological, and computational support for the hypothesis that defocusing attention results in divergent or associative thought, conducive to insight and finding unusual connections, while focusing attention results in convergent or analytic thought, conducive to rulebased operations. Creativity appears to involve both. It is widely believed that it is possible to escape mental fixation by spontaneously and temporarily engaging in a more divergent mode of thought. The resulting insight (if found) may be refined in a more analytic mode of thought. The questions addressed here are: (1) how does the architecture of memory support these two modes of thought, and (2) what is happening at the neural level when one shifts between them? Recent advances in neuroscience shed light on this. Activated cell assemblies are composed of multiple neural cliques, groups of neurons that respond differentially to general or contextspecific aspects of a situation. I refer to neural cliques that would not be included in the assembly if one were in an analytic mode, but would be if one were in an associative mode, as neurds. It is posited that the shift to a more associative mode of thought is accomplished by recruiting neurds that respond to abstract or atypical
On the Impact of Forgetting on Learning Machines
 Journal of the ACM
, 1993
"... this paper contributes toward the goal of understanding how a computer can be programmed to learn by isolating features of incremental learning algorithms that theoretically enhance their learning potential. In particular, we examine the effects of imposing a limit on the amount of information that ..."
Abstract

Cited by 15 (5 self)
 Add to MetaCart
(Show Context)
this paper contributes toward the goal of understanding how a computer can be programmed to learn by isolating features of incremental learning algorithms that theoretically enhance their learning potential. In particular, we examine the effects of imposing a limit on the amount of information that learning algorithm can hold in its memory as it attempts to This work was facilitated by an international agreement under NSF Grant 9119540.
Catastrophic forgetting and the pseudorehearsal solution in Hopfieldtype networks
 Connection Science
, 1998
"... Pseudorehearsal is a mechanism proposed by Robins which alleviates catastrophic forgetting in multilayer perceptron networks. In this paper, we extend the exploration of pseudorehearsal to a Hop ® eldtype net. The same general principles apply: old information can be rehearsed if it is available, ..."
Abstract

Cited by 15 (5 self)
 Add to MetaCart
(Show Context)
Pseudorehearsal is a mechanism proposed by Robins which alleviates catastrophic forgetting in multilayer perceptron networks. In this paper, we extend the exploration of pseudorehearsal to a Hop ® eldtype net. The same general principles apply: old information can be rehearsed if it is available, and if it is not available, then generating and rehearsing approximations of old information that `map ’ the behaviour of the network can also be eŒective at preser ving the actual old information itself. The details of the pseudorehearsal mechanism, however, bene ® t from being adapted to the dynamics of Hop ® eld nets so as to exploit the extra attractors created in state space during learning. These attractors are usually described as `spurious ’ or `crosstalk’, and regarded as undesirable, interfering with the retention of the trained population items. Our simulations have shown that, in another sense, such attractors can in fact be useful in preser ving the learned population. In general terms, a solution to the catastrophic forgetting problem enables the ongoing or sequential learning of information in arti ® cial neural networks, and consequently also provides a framework for the modelling of lifelong learning/developmental eŒects in cognition.