Results 1  10
of
54
Noisy Time Series Prediction using a Recurrent Neural Network and Grammatical Inference
 Machine Learning
, 2001
"... Financial forecasting is an example of a signal processing problem which is challenging due to small sample sizes, high noise, nonstationarity, and nonlinearity. Neural networks have been very successful in a number of signal processing applications. We discuss fundamental limitations and inherent ..."
Abstract

Cited by 49 (0 self)
 Add to MetaCart
(Show Context)
Financial forecasting is an example of a signal processing problem which is challenging due to small sample sizes, high noise, nonstationarity, and nonlinearity. Neural networks have been very successful in a number of signal processing applications. We discuss fundamental limitations and inherent difficulties when using neural networks for the processing of high noise, small sample size signals. We introduce a new intelligent signal processing method which addresses the difficulties. The method proposed uses conversion into a symbolic representation with a selforganizing map, and grammatical inference with recurrent neural networks. We apply the method to the prediction of daily foreign exchange rates, addressing difficulties with nonstationarity, overfitting, and unequal a priori class probabilities, and we find significant predictability in comprehensive experiments covering 5 different foreign exchange rates. The method correctly predicts the direction of change for th...
Architectural Bias in Recurrent Neural Networks  Fractal Analysis
 IEEE TRANSACTIONS ON NEURAL NETWORKS
"... We have recently shown that when initialized with "small" weights, recurrent neural networks (RNNs) with standard sigmoidtype activation functions are inherently biased towards Markov models, i.e. even prior to any training, RNN dynamics can be readily used to extract finite memory machin ..."
Abstract

Cited by 38 (8 self)
 Add to MetaCart
(Show Context)
We have recently shown that when initialized with "small" weights, recurrent neural networks (RNNs) with standard sigmoidtype activation functions are inherently biased towards Markov models, i.e. even prior to any training, RNN dynamics can be readily used to extract finite memory machines (Hammer & Tino, 2002; Tino, Cernansky & Benuskova, 2002; Tino, Cernansky & Benuskova, 2002a). Following Christiansen and Chater (1999), we refer to this phenomenon as the architectural bias of RNNs. In this paper we further extend our work on the architectural bias in RNNs by performing a rigorous fractal analysis of recurrent activation patterns. We assume the network is driven by sequences obtained by traversing an underlying finitestate transition diagram  a scenario that has been frequently considered in the past e.g. when studying RNNbased learning and implementation of regular grammars and finitestate transducers. We obtain lower and upper bounds on various types of fractal dimensions, such as boxcounting and Hausdorff dimensions. It turns out that not only can the recurrent activations inside RNNs with small initial weights be explored to build Markovian predictive models, but also the activations form fractal clusters the dimension of which can be bounded by the scaled entropy of the underlying driving source. The scaling factors are fixed and are given by the RNN parameters.
Rule Extraction from Recurrent Neural Networks: a Taxonomy and Review
 Neural Computation
, 2005
"... this paper, the progress of this development is reviewed and analysed in detail. In order to structure the survey and to evaluate the techniques, a taxonomy, specifically designed for this purpose, has been developed. Moreover, important open research issues are identified, that, if addressed pr ..."
Abstract

Cited by 27 (5 self)
 Add to MetaCart
this paper, the progress of this development is reviewed and analysed in detail. In order to structure the survey and to evaluate the techniques, a taxonomy, specifically designed for this purpose, has been developed. Moreover, important open research issues are identified, that, if addressed properly, possibly can give the field a significant push forward
Incremental Syntactic Parsing of Natural Language Corpora with Simple Synchrony Networks
 IEEE Transactions on Knowledge and Data Engineering
, 2001
"... This article explores the use of Simple Synchrony Networks (SSNs) for learning to parse English sentences drawn from a corpus of naturally occurring text. Parsing natural language sentences requires taking a sequence of words and outputting a hierarchical structure representing how those words fi ..."
Abstract

Cited by 25 (4 self)
 Add to MetaCart
(Show Context)
This article explores the use of Simple Synchrony Networks (SSNs) for learning to parse English sentences drawn from a corpus of naturally occurring text. Parsing natural language sentences requires taking a sequence of words and outputting a hierarchical structure representing how those words fit together to form constituents. Feedforward and Simple Recurrent Networks have had great difficulty with this task, in part because the number of relationships required to specify a structure is too large for the number of unit outputs they have available. SSNs have the representational power to output the necessary O(n 2 ) possible structural relationships, because SSNs extend the O(n) incremental outputs of Simple Recurrent Networks with the O(n) entity outputs provided by Temporal Synchrony Variable Binding. This article presents an incremental representation of constituent structures which allows SSNs to make effective use of both these dimensions. Experiments on learning to ...
Grammar Inference, Automata Induction, and Language Acquisition
 Handbook of Natural Language Processing
, 2000
"... The natural language learning problem has attracted the attention of researchers for several decades. Computational and formal models of language acquisition have provided some preliminary, yet promising insights of how children learn the language of their community. Further, these formal models als ..."
Abstract

Cited by 22 (1 self)
 Add to MetaCart
(Show Context)
The natural language learning problem has attracted the attention of researchers for several decades. Computational and formal models of language acquisition have provided some preliminary, yet promising insights of how children learn the language of their community. Further, these formal models also provide an operational framework for the numerous practical applications of language learning. We will survey some of the key results in formal language learning. In particular, we will discuss the prominent computational approaches for learning different classes of formal languages and discuss how these fit in the broad context of natural language learning.
Learning to predict a contextfree language: Analysis of dynamics in recurrent hidden units
, 1999
"... Recurrent neural network processing of regular language is reasonably well understood. Recent work has examined the less familiar question of contextfree languages. Previous results regarding the language a suggest that while it is possible for a small recurrent network to process contextfree ..."
Abstract

Cited by 20 (11 self)
 Add to MetaCart
Recurrent neural network processing of regular language is reasonably well understood. Recent work has examined the less familiar question of contextfree languages. Previous results regarding the language a suggest that while it is possible for a small recurrent network to process contextfree languages, learning them is di#cult. This paper considers the reasons underlying this di#culty by considering the relationship between the dynamics of the network and weightspace. We are able to show that the dynamics required for the solution lie in a region of weightspace close to a bifurcation point where small changes in weights may result in radically di#erent network behaviour. Furthermore, we show that the error gradient information in this region is highly irregular. We conclude that any gradientbased learning method will experience di#culty in learning the language due to the nature of the space, and that a more promising approach to improving learning performance may be to make weight changes in a nonindependent manner.
Incremental Nonmonotonic Parsing through Semantic SelfOrganization
 In Proc. of the 25th Annual Conf. of the Cognitive Science Society, Mahwah, NJ
, 2003
"... Subsymbolic systems have been successfully used to model several aspects of human language processing. Subsymbolic parsers are appealing because they allow combining syntactic, semantic, and thematic constraints in sentence interpretation and revising that interpretation as each word is read in. The ..."
Abstract

Cited by 8 (2 self)
 Add to MetaCart
Subsymbolic systems have been successfully used to model several aspects of human language processing. Subsymbolic parsers are appealing because they allow combining syntactic, semantic, and thematic constraints in sentence interpretation and revising that interpretation as each word is read in. These parsers are also cognitively plausible: processing is robust and multiple interpretations are simultaneously activated when the input is ambiguous. Yet, it has been very difficult to scale them up to realistic language. They have limited memory capacity, training takes a long time, and it is difficult to represent linguistic structure. In this study, we propose to scale up the subsymbolic approach by utilizing semantic selforganization. The resulting architecture, INSOMNET, was trained on semantic representations of the newlyreleased LINGO Redwoods HPSG Treebank of annotated sentences from the VerbMobil project. The results show that INSOMNET is able to accurately represent the semantic dependencies while demonstrating expectations and defaults, coactivation of multiple interpretations, and robust parsing of noisy input.
Learning Beyond Finite Memory in Recurrent Networks of Spiking Neurons
 Advances in Natural Computation  ICNC 2005, Lecture Notes in Computer Science
, 2005
"... Abstract. We investigate possibilities of inducing temporal structures without fading memory in recurrent networks of spiking neurons strictly operating in the pulsecoding regime. We extend the existing gradientbased algorithm for training feedforward spiking neuron networks (SpikeProp [1]) to re ..."
Abstract

Cited by 5 (0 self)
 Add to MetaCart
(Show Context)
Abstract. We investigate possibilities of inducing temporal structures without fading memory in recurrent networks of spiking neurons strictly operating in the pulsecoding regime. We extend the existing gradientbased algorithm for training feedforward spiking neuron networks (SpikeProp [1]) to recurrent network topologies, so that temporal dependencies in the input stream are taken into account. It is shown that temporal structures with unbounded input memory specified by simple Moore machines (MM) can be induced by recurrent spiking neuron networks (RSNN). The networks are able to discover pulsecoded representations of abstract information processing states coding potentially unbounded histories of processed inputs. 1
Recurrent Neural Networks for Time Series Classification
, 2003
"... Recurrent neural networks (RNN) are a widely used tool for the prediction of time series. In this paper we use the dynamic behaviour of the RNN to categorize input sequences into different specified classes. These two tasks do not seem to have much in common. However, the prediction task strongly su ..."
Abstract

Cited by 5 (0 self)
 Add to MetaCart
Recurrent neural networks (RNN) are a widely used tool for the prediction of time series. In this paper we use the dynamic behaviour of the RNN to categorize input sequences into different specified classes. These two tasks do not seem to have much in common. However, the prediction task strongly supports the development of a suitable internal structure, representing the main features of the input sequence, to solve the classification problem. Therefore, the speed and success of the training as well as the generalization ability of the trained RNN are significantly improved. The trained RNN provides good classification performance and enables the user to assess efficiently the degree of reliability of the classification result.