Results 1  10
of
44
Finding structure in time
 COGNITIVE SCIENCE
, 1990
"... Time underlies many interesting human behaviors. Thus, the question of how to represent time in connectionist models is very important. One approach is to represent time implicitly by its effects on processing rather than explicitly (as in a spatial representation). The current report develops a pro ..."
Abstract

Cited by 1573 (22 self)
 Add to MetaCart
Time underlies many interesting human behaviors. Thus, the question of how to represent time in connectionist models is very important. One approach is to represent time implicitly by its effects on processing rather than explicitly (as in a spatial representation). The current report develops a proposal along these lines first described by Jordan (1986) which involves the use of recurrent links in order to provide networks with a dynamic memory. In this approach, hidden unit patterns are fed back to themselves; the internal representations which develop thus reflect task demands in the context of prior internal states. A set of simulations is reported which range from relatively simple problems (temporal version of XOR) to discovering syntactic/semantic features for words. The networks are able to learn interesting internal representations which incorporate task demands with memory demands; indeed, in this approach the notion of memory is inextricably bound up with task processing. These representations reveal a rich structure, which allows them to be highly contextdependent while also expressing generalizations across classes of items. These representations suggest a method for representing lexical categories and the type/token distinction.
Connectionist Learning Procedures
 ARTIFICIAL INTELLIGENCE
, 1989
"... A major goal of research on networks of neuronlike processing units is to discover efficient learning procedures that allow these networks to construct complex internal representations of their environment. The learning procedures must be capable of modifying the connection strengths in such a way ..."
Abstract

Cited by 341 (6 self)
 Add to MetaCart
A major goal of research on networks of neuronlike processing units is to discover efficient learning procedures that allow these networks to construct complex internal representations of their environment. The learning procedures must be capable of modifying the connection strengths in such a way that internal units which are not part of the input or output come to represent important features of the task domain. Several interesting gradientdescent procedures have recently been discovered. Each connection computes the derivative, with respect to the connection strength, of a global measure of the error in the performance of the network. The strength is then adjusted in the direction that decreases the error. These relatively simple, gradientdescent learning procedures work well for small tasks and the new challenge is to find ways of improving their convergence rate and their generalization abilities so that they can be applied to larger, more realistic tasks.
Neural Net Architectures for Temporal Sequence Processing
, 1994
"... I present a general taxonomy of neural net architectures for processing timevarying patterns. This taxonomy subsumes many existing architectures in the literature, and points to several promising architectures that have yet to be examined. Any architecture that processes timevarying patterns requir ..."
Abstract

Cited by 107 (0 self)
 Add to MetaCart
I present a general taxonomy of neural net architectures for processing timevarying patterns. This taxonomy subsumes many existing architectures in the literature, and points to several promising architectures that have yet to be examined. Any architecture that processes timevarying patterns requires two conceptually distinct components: a shortterm memory that holds on to relevant past events and an associator that uses the shortterm memory to classify or predict. My taxonomy is based on a characterization of shortterm memory models along the dimensions of form, content, and adaptability. Experiments on predicting future values of a financial time series (US dollarSwiss franc exchange rates) are presented using several alternative memory models. The results of these experiments serve as a baseline against which more sophisticated architectures can be compared. Neural networks have proven to be a promising alternative to traditional techniques for nonlinear temporal prediction t...
Spatial and temporal pattern analysis via spiking neurons
 Network: Comput. Neural Syst
, 1998
"... ..."
The Theory of Discrete Lagrange Multipliers for Nonlinear Discrete Optimization
 Principles and Practice of Constraint Programming
, 1999
"... In this paper we present a Lagrangemultiplier formulation of discrete constrained optimization problems, the associated discretespace firstorder necessary and sufficient conditions for saddle points, and an efficient firstorder search procedure that looks for saddle points in discrete space. Our ..."
Abstract

Cited by 38 (21 self)
 Add to MetaCart
In this paper we present a Lagrangemultiplier formulation of discrete constrained optimization problems, the associated discretespace firstorder necessary and sufficient conditions for saddle points, and an efficient firstorder search procedure that looks for saddle points in discrete space. Our new theory provides a strong mathematical foundation for solving general nonlinear discrete optimization problems. Specifically, we propose a new vectorbased definition of descent directions in discrete space and show that the new definition does not obey the rules of calculus in continuous space. Starting from the concept of saddle points and using only vector calculus, we then prove the discretespace firstorder necessary and sufficient conditions for saddle points. Using welldefined transformations on the constraint functions, we further prove that the set of discretespace saddle points is the same as the set of constrained local minima, leading to the firstorder necessary and sufficient conditions for constrained local minima. Based on the firstorder conditions, we propose a localsearch method to look for saddle points that satisfy the firstorder conditions.
Speech Recognition using Neural Networks
, 1995
"... This thesis examines how artificial neural networks can benefit a large vocabulary, speaker independent, continuous speech recognition system. Currently, most speech recognition systems are based on hidden Markov models (HMMs), a statistical framework that supports both acoustic and temporal modelin ..."
Abstract

Cited by 30 (0 self)
 Add to MetaCart
This thesis examines how artificial neural networks can benefit a large vocabulary, speaker independent, continuous speech recognition system. Currently, most speech recognition systems are based on hidden Markov models (HMMs), a statistical framework that supports both acoustic and temporal modeling. Despite their stateoftheart performance, HMMs make a number of suboptimal modeling assumptions that limit their potential effectiveness. Neural networks avoid many of these assumptions, while they can also learn complex functions, generalize effectively, tolerate noise, and support parallelism. While neural networks can readily be applied to acoustic modeling, it is not yet clear how they can be used for temporal modeling. Therefore, we explore a class of systems called NNHMM hybrids, in which neural networks perform acoustic modeling, and HMMs perform temporal modeling. We argue that a NNHMM hybrid has several theoretical advantages over a pure HMM system, including better acoustic ...
Primitive Auditory Segregation Based On Oscillatory Correlation
 Cognitive Science
, 1996
"... Auditory scene analysis is critical for complex auditory processing. We study auditory segregation from the neural network perspective, and develop a framework for primitive auditory scene analysis. The architecture is a laterally coupled twodimensional network of relaxation oscillators with a glob ..."
Abstract

Cited by 26 (7 self)
 Add to MetaCart
Auditory scene analysis is critical for complex auditory processing. We study auditory segregation from the neural network perspective, and develop a framework for primitive auditory scene analysis. The architecture is a laterally coupled twodimensional network of relaxation oscillators with a global inhibitor. One dimension represents time and another one represents frequency. We show that this architecture, plus systematic delay lines, can in real time group auditory features into a stream by phase synchrony and segregate different streams by desynchronization. The network demonstrates a set of psychological phenomena regarding primitive auditory scene analysis, including dependency on frequency proximity and the rate of presentation, sequential capturing, and competition among different perceptual organizations. We offer a neurocomputational theory  shifting synchronization theory  for explaining how auditory segregation might be achieved in the brain, and the psychological pheno...
Neural Network Models of Sensory Integration for Improved Vowel Recognition
, 1990
"... Automatic speech recognizers currently perform poorly in the presence of noise. Humans, on the other hand, often compensate for noise degradation by extracting speech information from alternative sources and then integrating this information with the acoustical signal. Visual signals from the speake ..."
Abstract

Cited by 23 (2 self)
 Add to MetaCart
Automatic speech recognizers currently perform poorly in the presence of noise. Humans, on the other hand, often compensate for noise degradation by extracting speech information from alternative sources and then integrating this information with the acoustical signal. Visual signals from the speaker’s face are one source of supplemental speech information. We demonstrate that multiple sources of speech information can be integrated at a subsymbolic level to improve vowel recognition. Feedforward and recurrent neural networks are trained to estimate the acoustic characteristics of the vocal tract from images of the speaker‘s mouth. These estimates are then combined with the noisedegraded acoustic information, effectively increasing the signaltonoise ratio and improving the recognition of these noisedegraded signals. Alternative symbolic strategies, such as direct categorization of the visual signals into vowels, are also presented. The performances of these neural networks compared favorably with human performance and with other patternmatching and estimation techniques.
Relationship between afferent and central temporal patterns in the locust olfactory system
 The Journal of Neuroscience
, 1999
"... ..."
A Neural Network Model of Temporal Code Generation and PositionInvariant Pattern Recognition
, 1998
"... Numerous studies have suggested that the brain may encode information in the temporal firing pattern of neurons. However, little is known... ..."
Abstract

Cited by 12 (1 self)
 Add to MetaCart
Numerous studies have suggested that the brain may encode information in the temporal firing pattern of neurons. However, little is known...