Results 1  10
of
30
A Guide to the Literature on Learning Probabilistic Networks From Data
, 1996
"... This literature review discusses different methods under the general rubric of learning Bayesian networks from data, and includes some overlapping work on more general probabilistic networks. Connections are drawn between the statistical, neural network, and uncertainty communities, and between the ..."
Abstract

Cited by 190 (0 self)
 Add to MetaCart
This literature review discusses different methods under the general rubric of learning Bayesian networks from data, and includes some overlapping work on more general probabilistic networks. Connections are drawn between the statistical, neural network, and uncertainty communities, and between the different methodological communities, such as Bayesian, description length, and classical statistics. Basic concepts for learning and Bayesian networks are introduced and methods are then reviewed. Methods are discussed for learning parameters of a probabilistic network, for learning the structure, and for learning hidden variables. The presentation avoids formal definitions and theorems, as these are plentiful in the literature, and instead illustrates key concepts with simplified examples. Keywords Bayesian networks, graphical models, hidden variables, learning, learning structure, probabilistic networks, knowledge discovery. I. Introduction Probabilistic networks or probabilistic gra...
A Recurrent Neural Network That Learns to Count
 CONNECTION SCIENCE
, 1999
"... ..."
(Show Context)
Architectural Bias in Recurrent Neural Networks  Fractal Analysis
 IEEE TRANSACTIONS ON NEURAL NETWORKS
"... We have recently shown that when initialized with "small" weights, recurrent neural networks (RNNs) with standard sigmoidtype activation functions are inherently biased towards Markov models, i.e. even prior to any training, RNN dynamics can be readily used to extract finite memory machin ..."
Abstract

Cited by 42 (9 self)
 Add to MetaCart
(Show Context)
We have recently shown that when initialized with "small" weights, recurrent neural networks (RNNs) with standard sigmoidtype activation functions are inherently biased towards Markov models, i.e. even prior to any training, RNN dynamics can be readily used to extract finite memory machines (Hammer & Tino, 2002; Tino, Cernansky & Benuskova, 2002; Tino, Cernansky & Benuskova, 2002a). Following Christiansen and Chater (1999), we refer to this phenomenon as the architectural bias of RNNs. In this paper we further extend our work on the architectural bias in RNNs by performing a rigorous fractal analysis of recurrent activation patterns. We assume the network is driven by sequences obtained by traversing an underlying finitestate transition diagram  a scenario that has been frequently considered in the past e.g. when studying RNNbased learning and implementation of regular grammars and finitestate transducers. We obtain lower and upper bounds on various types of fractal dimensions, such as boxcounting and Hausdorff dimensions. It turns out that not only can the recurrent activations inside RNNs with small initial weights be explored to build Markovian predictive models, but also the activations form fractal clusters the dimension of which can be bounded by the scaled entropy of the underlying driving source. The scaling factors are fixed and are given by the RNN parameters.
Rule Extraction from Recurrent Neural Networks: a Taxonomy and Review
 Neural Computation
, 2005
"... this paper, the progress of this development is reviewed and analysed in detail. In order to structure the survey and to evaluate the techniques, a taxonomy, specifically designed for this purpose, has been developed. Moreover, important open research issues are identified, that, if addressed pr ..."
Abstract

Cited by 35 (5 self)
 Add to MetaCart
this paper, the progress of this development is reviewed and analysed in detail. In order to structure the survey and to evaluate the techniques, a taxonomy, specifically designed for this purpose, has been developed. Moreover, important open research issues are identified, that, if addressed properly, possibly can give the field a significant push forward
ContextFree and ContextSensitive Dynamics in Recurrent Neural Networks
, 2000
"... Continuousvalued recurrent neural networks can learn mechanisms for processing contextfree languages. The dynamics of such networks is usually based on damped oscillation around fixed points in state space and requires that the dynamical components are arranged in certain ways. It is shown tha ..."
Abstract

Cited by 34 (7 self)
 Add to MetaCart
Continuousvalued recurrent neural networks can learn mechanisms for processing contextfree languages. The dynamics of such networks is usually based on damped oscillation around fixed points in state space and requires that the dynamical components are arranged in certain ways. It is shown that qualitatively similar dynamics with similar constraints hold for a n b n c n , a contextsensitive language. The additional difficulty with a n b n c n , compared with the contextfree language a n b n , consists of "counting up" and "counting down" letters simultaneously. The network solution is to oscillate in two principal dimensions, one for counting up and one for counting down. This study focuses on the dynamics employed by the Sequential Cascaded Network, in contrast with the Simple Recurrent Network, and the use of Backpropagation Through Time. Found solutions generalize well beyond training data, however, learning is not reliable. The contribution of this ...
A Guide to Recurrent Neural Networks and Backpropagation
 IN THE DALLAS PROJECT, SICS TECHNICAL REPORT T2002:03, SICS
, 2002
"... This paper provides guidance to some of the concepts surrounding recurrent neural networks. Contrary to feedforward networks, recurrent networks can be sensitive, and be adapted to past inputs. Backpropagation learning is described for feedforward networks, adapted to suit our (probabilistic) modeli ..."
Abstract

Cited by 18 (1 self)
 Add to MetaCart
This paper provides guidance to some of the concepts surrounding recurrent neural networks. Contrary to feedforward networks, recurrent networks can be sensitive, and be adapted to past inputs. Backpropagation learning is described for feedforward networks, adapted to suit our (probabilistic) modeling needs, and extended to cover recurrent networks. The aim of this brief paper is to set the scene for applying and understanding recurrent neural networks.
Knowledge Extraction from Transducer Neural Networks
 Journal of Applied Intelligence
, 2000
"... Previously neural networks have shown interesting performance results for tasks such as classification, but they still suffer from an insufficient focus on the structure of the knowledge represented therein. In this paper, we analyze various knowledge extraction techniques in detail and we develop n ..."
Abstract

Cited by 13 (5 self)
 Add to MetaCart
Previously neural networks have shown interesting performance results for tasks such as classification, but they still suffer from an insufficient focus on the structure of the knowledge represented therein. In this paper, we analyze various knowledge extraction techniques in detail and we develop new transducer extraction techniques for the interpretation of recurrent neural network learning. First, we provide an overview of different possibilities to express structured knowledge using neural networks. Then, we analyze a type of recurrent network rigorously, applying a broad range of different techniques. We argue that analysis techniques, such as weight analysis using Hinton diagrams, hierarchical cluster analysis, and principal component analysis may be useful for providing certain views on the underlying knowledge. However, we demonstrate that these techniques are too static and too lowlevel for interpreting recurrent network classifications. The contribution of this paper is a particularly broad analysis of knowledge extraction techniques. Furthermore, we propose dynamic learning analysis and transducer extraction as two new dynamic interpretation techniques. Dynamic learning analysis provides a better understanding of how the network learns, while transducer extraction provides a better understanding of what the network represents.
Identification of Behaviors in an Agent's Phase Space
, 1995
"... This paper describes a method for analyzing the behavior of an autonomous robot. The robot is viewed as a continuous# stochastic dynamical system. The analysis starts from an empirical phase portrait. In a first stage, elementary regularities are detected. These regularities, called transient attrac ..."
Abstract

Cited by 10 (5 self)
 Add to MetaCart
This paper describes a method for analyzing the behavior of an autonomous robot. The robot is viewed as a continuous# stochastic dynamical system. The analysis starts from an empirical phase portrait. In a first stage, elementary regularities are detected. These regularities, called transient attractors, combine properties of attractors with properties of partition cells of phase space. As the system trajectory passes through these regularities repeatedly, a sequence of identifiable events is produced, which can be interpreted as a symbol sequence. This sequence is further analyzed in the second stage, where a finite description of temporal regularities within it is constructed. This description comes in the format of a variety of finitestate automata. In a third stage, higherorder regularities in this finitestatelike description are identified. This yields a hierarchic behavior model. At all stages, regularities are defined using criteria of maximal local predictability. Thus, the enti...
Attractive Periodic Sets in DiscreteTime Recurrent Networks (with Emphasis on FixedPoint Stability and Bifurcations in TwoNeuron Networks)
, 2001
"... We perform a detailed fixedpoint analysis of twounit recurrent neural networks with sigmoidshaped transfer functions. Using geometrical arguments in the space of transfer function derivatives, we partition the network statespace into distinct regions corresponding to stability types of the fixed ..."
Abstract

Cited by 8 (0 self)
 Add to MetaCart
We perform a detailed fixedpoint analysis of twounit recurrent neural networks with sigmoidshaped transfer functions. Using geometrical arguments in the space of transfer function derivatives, we partition the network statespace into distinct regions corresponding to stability types of the fixed points. Unlike in the previous studies, we do not assume any special form of connectivity pattern between the neurons, and all free parameters are allowed to vary. We also prove that when both neurons have excitatory selfconnections and the mutual interaction pattern is the same (i.e., the neurons mutually inhibit or excite themselves), new attractive fixed points are created through the saddlenode bifurcation. Finally, for an Nneuron recurrent network, we give lower bounds on the rate of convergence of attractive periodic points toward the saturation values of neuron activations, as the absolute values of connection weights grow.
Fixed Points in TwoNeuron Discrete Time Recurrent Networks: Stability and Bifurcation Considerations
, 1995
"... The position, number and stability types of fixed points of a twoneuron recurrent network with nonzero weights are investigated. Using simple geometrical arguments in the space of derivatives of the sigmoid transfer function with respect to the weighted sum of neuron inputs, we partition the netwo ..."
Abstract

Cited by 8 (1 self)
 Add to MetaCart
(Show Context)
The position, number and stability types of fixed points of a twoneuron recurrent network with nonzero weights are investigated. Using simple geometrical arguments in the space of derivatives of the sigmoid transfer function with respect to the weighted sum of neuron inputs, we partition the network state space into several regions corresponding to stability types of the fixed points. If the neurons have the same mutual interaction pattern, i.e. they either mutually inhibit or mutually excite themselves, a lower bound on the rate of convergence of the attractive fixed points towards the saturation values, as the absolute values of weights on the selfloops grow, is given. The role of weights in location of fixed points is explored through an intuitively appealing characterization of neurons according to their inhibition/excitation performance in the network. In particular, each neuron can be of one of the four types: greedy, enthusiastic, altruistic or depressed. Both with and witho...