Results 1  10
of
14
Finding structure in time
 COGNITIVE SCIENCE
, 1990
"... Time underlies many interesting human behaviors. Thus, the question of how to represent time in connectionist models is very important. One approach is to represent time implicitly by its effects on processing rather than explicitly (as in a spatial representation). The current report develops a pro ..."
Abstract

Cited by 1533 (21 self)
 Add to MetaCart
Time underlies many interesting human behaviors. Thus, the question of how to represent time in connectionist models is very important. One approach is to represent time implicitly by its effects on processing rather than explicitly (as in a spatial representation). The current report develops a proposal along these lines first described by Jordan (1986) which involves the use of recurrent links in order to provide networks with a dynamic memory. In this approach, hidden unit patterns are fed back to themselves; the internal representations which develop thus reflect task demands in the context of prior internal states. A set of simulations is reported which range from relatively simple problems (temporal version of XOR) to discovering syntactic/semantic features for words. The networks are able to learn interesting internal representations which incorporate task demands with memory demands; indeed, in this approach the notion of memory is inextricably bound up with task processing. These representations reveal a rich structure, which allows them to be highly contextdependent while also expressing generalizations across classes of items. These representations suggest a method for representing lexical categories and the type/token distinction.
Biologically Plausible Errordriven Learning using Local Activation Differences: The Generalized Recirculation Algorithm
 NEURAL COMPUTATION
, 1996
"... The error backpropagation learning algorithm (BP) is generally considered biologically implausible because it does not use locally available, activationbased variables. A version of BP that can be computed locally using bidirectional activation recirculation (Hinton & McClelland, 1988) instead of ..."
Abstract

Cited by 94 (10 self)
 Add to MetaCart
The error backpropagation learning algorithm (BP) is generally considered biologically implausible because it does not use locally available, activationbased variables. A version of BP that can be computed locally using bidirectional activation recirculation (Hinton & McClelland, 1988) instead of backpropagated error derivatives is more biologically plausible. This paper presents a generalized version of the recirculation algorithm (GeneRec), which overcomes several limitations of the earlier algorithm by using a generic recurrent network with sigmoidal units that can learn arbitrary input/output mappings. However, the contrastiveHebbian learning algorithm (CHL, a.k.a. DBM or mean field learning) also uses local variables to perform errordriven learning in a sigmoidal recurrent network. CHL was derived in a stochastic framework (the Boltzmann machine), but has been extended to the deterministic case in various ways, all of which rely on problematic approximationsand assumptions, le...
Recurrent neural networks and robust time series prediction
 IEEE TRANSACTIONS ON NEURAL NETWORKS
, 1994
"... We propose a robust learning algorithm and apply it to recurrent neural networks. This algorithm is based on filtering outliers from the data and then estimating parameters from the filtered data. The filtering removes outliers from both the target function and the inputs of the neural network. The ..."
Abstract

Cited by 48 (2 self)
 Add to MetaCart
We propose a robust learning algorithm and apply it to recurrent neural networks. This algorithm is based on filtering outliers from the data and then estimating parameters from the filtered data. The filtering removes outliers from both the target function and the inputs of the neural network. The filtering is soff in that some outliers are neither completely rejected nor accepted. To show the need for robust recurrent networks, we compare the predictive ability of least squares estimated recurrent networks on synthetic data and on the Puget Power Electric Demand time series. These investigations result in a class of recurrent neural networks, NARMA(p, q), which show advantages over feedforward neural networks for time series with a moving average component. Conventional least squares methods of fitting NARMA(p,q) neural network models are shown to suffer a lack of robustness towards outliers. This sensitivity to outliers is demonstrated on both the synthetic and real data sets. Filtering the Puget Power Electric Demand time series is shown to automatically remove the outliers due to holidays. Neural networks trained on filtered data are then shown to give better predictions than neural networks trained on unfiltered time series.
Generalization in Interactive Networks: The Benefits of Inhibitory Competition and Hebbian Learning
 Neural Computation
, 2001
"... Computational models in cognitive neuroscience should ideally use biological properties and powerful computational principles to produce behavior consistent with psychological findings. Errordriven backpropagation is computationally powerful, and has proven useful for modeling a range of psycholo ..."
Abstract

Cited by 45 (5 self)
 Add to MetaCart
Computational models in cognitive neuroscience should ideally use biological properties and powerful computational principles to produce behavior consistent with psychological findings. Errordriven backpropagation is computationally powerful, and has proven useful for modeling a range of psychological data, but is not biologically plausible. Several approaches to implementing backpropagation in a biologically plausible fashion converge on the idea of using bidirectional activation propagation in interactive networks to convey error signals. This paper demonstrates two main points about these errordriven interactive networks: (a) they generalize poorly due to attractor dynamics that interfere with the network's ability to systematically produce novel combinatorial representations in response to novel inputs; and (b) this generalization problem can be remedied by adding two widely used mechanistic principles, inhibitory competition and Hebbian learning, that can be independent...
Lending Direction to Neural Networks
 Neural Networks
, 1995
"... We present a general formulation for a network of stochastic directional units. This formulation is an extension of the Boltzmann machine in which the units are not binary, but take on values on a cyclic range, between 0 and 2ß radians. This measure is appropriate to many domains, representing cycli ..."
Abstract

Cited by 11 (3 self)
 Add to MetaCart
We present a general formulation for a network of stochastic directional units. This formulation is an extension of the Boltzmann machine in which the units are not binary, but take on values on a cyclic range, between 0 and 2ß radians. This measure is appropriate to many domains, representing cyclic or angular values, e.g., wind direction, days of the week, phases of the moon. The state of each unit in a DirectionalUnit Boltzmann Machine (DUBM) is described by a complex variable, where the phase component specifies a direction; the weights are also complex variables. We associate a quadratic energy function, and corresponding probability, with each DUBM configuration. The conditional distribution of a unit's stochastic state is a circular version of the Gaussian probability distribution, known as the von Mises distribution. In a meanfield approximation to a stochastic dubm, the phase component of a unit's state represents its mean direction, and the magnitude component specifies the...
The Simultaneous Recurrent Neural Network Addressing the Scaling Problem in Static Optimization
 International Journal of Neural Systems
, 2001
"... A trainable recurrent neural network, Simultaneous Recurrent Neural network, is proposed to address the scaling problem faced by neural network algorithms in static optimization. The proposed algorithm derives its computational power to address the scaling problem through its ability to "learn ..."
Abstract

Cited by 9 (6 self)
 Add to MetaCart
A trainable recurrent neural network, Simultaneous Recurrent Neural network, is proposed to address the scaling problem faced by neural network algorithms in static optimization. The proposed algorithm derives its computational power to address the scaling problem through its ability to "learn " compared to existing recurrent neural algorithms, which are not trainable. Recurrent backpropagation algorithm is employed to train the recurrent, relaxationbased neural network in order to associate fixed points of the network dynamics with locally optimal solutions of the static optimization problems. Performance of the algorithm is tested on the NPhard Traveling Salesman Problem in the range of 100 to 600 cities. Simulation results indicate that the proposed algorithm is able to consistently locate highquality solutions for all problem sizes tested. In other words, the proposed algorithm scales demonstrably well with the problem size with respect to quality of solutions and at the expense of increased computational cost for large problem sizes.
The Simultaneous Recurrent Neural Network for Static Optimization Problems
 The University of Toledo
, 1999
"... This paper presents a study on computational promise of Simultaneous Recurrent Networks to solve largescale optimization problems. Specifically the performance of the network for solving Traveling Salesman Problem is addressed and analyzed. A recurrent and trainable neural network, Simultaneous Rec ..."
Abstract

Cited by 2 (0 self)
 Add to MetaCart
This paper presents a study on computational promise of Simultaneous Recurrent Networks to solve largescale optimization problems. Specifically the performance of the network for solving Traveling Salesman Problem is addressed and analyzed. A recurrent and trainable neural network, Simultaneous Recurrent Network, with Recurrent Backpropagation training algorithm is employed to address difficulties related to scaling problem, which is currently hindering the successful application of Artificial Neural Network algorithms to largescale static optimization problems. The Simultaneous Recurrent Neural Network was successfully trained to locate "good quality " solutions to the Travelling Salesman Problem with up to 500 cities.
A Study on Inference Control in Natural Language Processing

, 1997
"... Natural language processing requires flexible control of computation on various sorts of constraints such as syntax, semantics, pragmatics. This study aims to propose and verify a new approach that describes a system declaratively with constraints and controls inferences guided by general principles ..."
Abstract

Cited by 2 (0 self)
 Add to MetaCart
Natural language processing requires flexible control of computation on various sorts of constraints such as syntax, semantics, pragmatics. This study aims to propose and verify a new approach that describes a system declaratively with constraints and controls inferences guided by general principles based on probability. This approach is an alternative one against the previous procedural approach that prepares a number of various heuristics on control. It is needed to use a various constraints flexibly not only in dialogue systems but also in the subsystem such as parser. In fact, [Nagata and Morimoto] and [Maxwell and Kaplan] pointed out that in modern grammatical formalisms such as HPSG and LFG, which employ two sorts of linguistic constraints (i.e. on phrase structures and feature structures), radical efficiency improvement could be obtained by appropriate strategy for combining computations on different constraints. Concretely speaking, processing relatively coarser grained constraints on phrase structure first and then other finer grained constraints such as on feature structures is more efficient in general. This phenomenon results from the difference in computational complexity between processing these sorts of constraints. That is, syntactic parsing on a context free grammar can be performed within the polynomial time and space with respect to the size of the input sentence while a unification of two feature
Dynamic Recurrent Neural Networks: a Dynamical Analysis
 IEEE TRANS. ON SYSTEMS MAN AND CYBERNETICS, PART B
, 1996
"... In this paper, we explore the dynamical features of a neural network model which presents two types of adaptative parameters : the classical weights between the units and the time constants associated with each artificial neuron. The purpose of this study is to provide a strong theoretical basis for ..."
Abstract

Cited by 2 (0 self)
 Add to MetaCart
In this paper, we explore the dynamical features of a neural network model which presents two types of adaptative parameters : the classical weights between the units and the time constants associated with each artificial neuron. The purpose of this study is to provide a strong theoretical basis for modeling and simulating dynamic recurrent neural networks. In order to achieve this, we study the effect of the statistical distribution of the weights and of the time constants on the network dynamics and we make a statistical analysis of the neural transformation. We examine the network power spectra (to draw some conclusions over the frequential behavior of the network) and we compute the stability regions to explore the stability of the model. We show that the network is sensitive to the variations of the mean values of the weights and the time constants (because of the temporal aspects of the learned tasks). Nevertheless, our results highlight the improvements in the network dynamics d...
Localist Attractor Networks
"... Attractor networks, which map an input space to a discrete output space, are useful for pattern completioncleaning up noisy or missing input features. However, designing a net to have a given set of attractors is notoriously tricky; training procedures are CPU intensive and often produce spuri ..."
Abstract

Cited by 2 (1 self)
 Add to MetaCart
Attractor networks, which map an input space to a discrete output space, are useful for pattern completioncleaning up noisy or missing input features. However, designing a net to have a given set of attractors is notoriously tricky; training procedures are CPU intensive and often produce spurious attractors and illconditioned attractor basins. These difficulties occur because each connection in the network participates in the encoding of multiple attractors. We describe an alternative formulation of attractor networks in which the encoding of knowledge is local, not distributed. Although localist attractor networks have similar dynamics to their distributed counterparts, they are much easier to work with and interpret. We propose a statistical formulation of localist attractor net dynamics, which yields a convergence proof and a mathematical interpretation of model parameters. We present simulation experiments that explore the behavior of localist attractor networks, show...