Time underlies many interesting human behaviors. Thus, the question of how to represent time in connectionist models is very important. One approach is to represent time implicitly by its effects on processing rather than explicitly (as in a spatial representation). The current report develops a proposal along these lines first described by Jordan (1986) which involves the use of recurrent links in order to provide networks with a dynamic memory. In this approach, hidden unit patterns are fed back to themselves; the internal representations which develop thus reflect task demands in the context of prior internal states. A set of simulations is reported which range from relatively simple problems (temporal version of XOR) to discovering syntactic/semantic features for words. The networks are able to learn interesting internal representations which incorporate task demands with memory demands; indeed, in this approach the notion of memory is inextricably bound up with task processing. These representations reveal a rich structure, which allows them to be highly context-dependent while also expressing generalizations across classes of items. These representations suggest a method for representing lexical categories and the type/token distinction.

I present a general taxonomy of neural net architectures for processing time-varying patterns. This taxonomy subsumes many existing architectures in the literature, and points to several promising architectures that have yet to be examined. Any architecture that processes timevarying patterns requires two conceptually distinct components: a short-term memory that holds on to relevant past events and an associator that uses the short-term memory to classify or predict. My taxonomy is based on a characterization of short-term memory models along the dimensions of form, content, and adaptability. Experiments on predicting future values of a financial time series (US dollar--Swiss franc exchange rates) are presented using several alternative memory models. The results of these experiments serve as a baseline against which more sophisticated architectures can be compared. Neural networks have proven to be a promising alternative to traditional techniques for nonlinear temporal prediction t...

The human reader of handwriting is unaware of the amount of back-ground knowledge that is constantly being used by a massive parallel computer, his brain, to decipher cursive script. Artificial cursive script recognizers do not have access to a comparable source of knowledge or of comparable computational power to perform top-down processing. Therefore, in an artificial script recognizer, there is a strong demand for reliable bottom-up processing. For the recognition of unrestricted script consisting of arbitrary character sequences, on-line recorded handwriting signals offer a more solid basis than the optically obtained grey-scale image of a written pen trace, because of the temporal information and the inherent vectorial description of shape. The enhanced bottom-up processing is based on implementing knowledge of the motor system in the handwriting recognition system. The bottom-up information will already be sufficient to recognize clearly written and unambiguous input. However, am...

In many real-world tasks, the ability to focus attention on the important features of the input is crucial for good performance. In this paper a mechanism for achieving task-specific focus of attention is presented. A saliency map, which is based upon a computed expectation of the contents of the inputs at the next time step, indicates which regions of the input retina are important for performing the task. The saliency map can be used to accentuate the features which are important, and de-emphasize those which are not. The performance of this method is demonstrated on a real-world robotics task: autonomous road following. The applicability of this method is also demonstrated in a non-visual domain. Architectural and algorithmic details are provided, as well as empirical results. Using the Representation in a Neural Network's Hidden Layer for Task-Specific Focus of Attention Shumeet Baluja & Dean Pomerleau May 22, 1995 CMU-CS-95-143 Shumeet Baluja is supported by a National Science Fo...

. Learning of temporal sequences is a topic of research in such different areas as speech and other temporal pattern recognition as well as motor control (for a survey see e.g. Mozer, 1993). In this paper an approach is presented which is particularly suitable for motor control due to the fact that it does not only reproduce temporal sequences in exactly the way they where learned but it is able to generate slightly modified sequences according to given parameters, too. Key words. Neural Networks, Temporal Sequences, Motor Control, Motor Programs. 1 Introduction The human motor system is based on the interaction of muscles which are controlled by specific parts of the nervous system. When examining the nervous system, one can recognize a hierarchical structure, physiologically as well as functionally. In the lowest level the spinal cord provides simple motion sequences commonly summarized as reflex actions. They are triggered either by sensory neurons or by signals from the higher le...

In this paper, we explore the dynamical features of a neural network model which presents two types of adaptative parameters : the classical weights between the units and the time constants associated with each artificial neuron. The purpose of this study is to provide a strong theoretical basis for modeling and simulating dynamic recurrent neural networks. In order to achieve this, we study the effect of the statistical distribution of the weights and of the time constants on the network dynamics and we make a statistical analysis of the neural transformation. We examine the network power spectra (to draw some conclusions over the frequential behavior of the network) and we compute the stability regions to explore the stability of the model. We show that the network is sensitive to the variations of the mean values of the weights and the time constants (because of the temporal aspects of the learned tasks). Nevertheless, our results highlight the improvements in the network dynamics d...

Data filtering methods are so much crucial to get good performance in time series forecasting. There are a few preprocessing methods (i.e. ARMA outputs as time domain filters, and Fourier transform or wavelet transform as time-frequency domain filters) for handling time series. In particular, the time-frequency domain filters describe the fractal structure of financial markets better than the time domain filters without frequency information. We study the issues of integrated methods of joint timefrequency analysis and neural network techniques as the application of multi-cyclic decomposition methods to the neural networks for short-term point forecast decisionmaking. The issues include the appropriate selection of neural network model architecture, for example, what type of neural network learning architecture is selected and what input size should be selected for our time series forecasting. We analyze these problems in particular with recurrent neural network learning and embedding dimension as chaos analysis. This study is also applied to a case study of daily Korean won / U.S. dollar exchange returns. Finally we suggest an integration framework for future research from our experimental results.

Heterogeneous types of gene expressions may provide a better insight into the biological role of gene interaction with the environment, disease development and drug effect at the molecular level. In this paper for both exploring and prediction purposes a Time Lagged Recurrent Neural Network with trajectory learning is proposed for identifying and classifying the gene functional patterns from the heterogeneous nonlinear time series microarray experiments. The proposed procedures identify gene functional patterns from the dynamics of a state-trajectory learned in the heterogeneous time series and the gradient information over time. Also, the trajectory learning with Backpropagation through time algorithm can recognize gene expression patterns vary over time. This may reveal much more information about the regulatory network underlying gene expressions. The analyzed data were extracted from spotted DNA microarrays in the budding yeast expression measurements, produced by Eisen et al. The gene matrix contained 79 experiments over a variety of heterogeneous experiment conditions. The number of recognized gene patterns in our study ranged from two to ten and were divided into three cases. Optimal network architectures with different memory structures were selected based on Akaike and Bayesian information statistical criteria using two-way factorial design. The optimal model performance was compared to other popular gene classification algorithms such as Nearest Neighbor, Support Vector Machine, and Self-Organized Map. The reliability of the performance was verified with multiple iterated runs.

The focused gamma network is proposed as one of the possible implementations of the gamma neural model. The focused gamma network is compared with the focused backpropagation network and TDNN for a time series prediction problem, and with ADALINE in a system identification problem. 1

Time is the center of many human tasks. To talk, to listen, to read or to write are examples of time related tasks. To integrate the time notion into neural network is very important in order to deal with such tasks. This report presents various tasks that are based on temporal pattern processing and the different neural network architectures, simulated to tackle the problem. We examine the main components of connectionist models that process time varying patterns: the memory that records past informations, the pattern of connectivity among units of the network, and the rule used to update connection strength during training. We explore two different network architectures, one presented by Elman [8] and the other by Stornetta et al. [19], and analyze their ability to learn and recognize a finite state machine. Variant of these architectures are explored in order to know whether better results may be reached or not. We finally compare the results obtained by these architectures for the ...