Abstract

Previous neural network learning algorithms for sequence processing are computationally expensive and perform poorly when it comes to long time lags. This paper first introduces a simple principle for reducing the descriptions of event sequences without loss of information. A consequence of this principle is that only unexpected inputs can be relevant. This insight leads to the construction of neural architectures that learn to `divide and conquer' by recursively decomposing sequences. I describe two architectures. The first functions as a self-organizing multi-level hierarchy of recurrent networks. The second, involving only two recurrent networks, tries to collapse a multi-level predictor hierarchy into a single recurrent net. Experiments show that the system can require less computation per time step and many fewer training sequences than conventional training algorithms for recurrent nets. 1 INTRODUCTION Several approaches to on-line supervised sequence learning have been propose...

Citations

105	An efficient gradient-based algorithm for on-line training of recurrent network trajectories - Williams, Peng - 1990
76	The utility driven dynamic error propagation network - Robinson, Fallside - 1987
20	Connectionist music composition based on melodic and stylistic constraints - Mozer, Soukup - 1991
19	Learning to control fast-weight memories: An alternative to dynamic recurrent networks., Neural Comput 4 - Schmidhuber - 1992
13	Adaptive decomposition of time - Schmidhuber - 1991
8	Neural sequence chunkers - Schmidhuber - 1991
6	The `moving targets' training method - Rohwer - 1989
5	An O(n ) learning algorithm for fully recurrent networks - Schmidhuber - 1991
3	An unsupervised PDP learning model for action planning - Miyata - 1988
1	Diploma thesis. Institut fur Informatik, Technische Universitat Munchen - Hochreiter - 1991