## Architectural Bias in Recurrent Neural Networks - Fractal Analysis (1931)

### Cached

### Download Links

Venue: | IEEE Transactions on Neural Networks |

Citations: | 28 - 7 self |

### BibTeX

@ARTICLE{Tino31architecturalbias,

author = {Peter Tino and Barbara Hammer},

title = {Architectural Bias in Recurrent Neural Networks - Fractal Analysis},

journal = {IEEE Transactions on Neural Networks},

year = {1931},

volume = {15},

pages = {6--15}

}

### Years of Citing Articles

### OpenURL

### Abstract

We have recently shown that when initialized with "small" weights, recurrent neural networks (RNNs) with standard sigmoid-type activation functions are inherently biased towards Markov models, i.e. even prior to any training, RNN dynamics can be readily used to extract finite memory machines (Hammer & Tino, 2002; Tino, Cernansky & Benuskova, 2002; Tino, Cernansky & Benuskova, 2002a). Following Christiansen and Chater (1999), we refer to this phenomenon as the architectural bias of RNNs. In this paper we further extend our work on the architectural bias in RNNs by performing a rigorous fractal analysis of recurrent activation patterns. We assume the network is driven by sequences obtained by traversing an underlying finite-state transition diagram -- a scenario that has been frequently considered in the past e.g. when studying RNN-based learning and implementation of regular grammars and finite-state transducers. We obtain lower and upper bounds on various types of fractal dimensions, such as box-counting and Hausdor# dimensions. It turns out that not only can the recurrent activations inside RNNs with small initial weights be explored to build Markovian predictive models, but also the activations form fractal clusters the dimension of which can be bounded by the scaled entropy of the underlying driving source. The scaling factors are fixed and are given by the RNN parameters.