Learning Precise Timing with LSTM Recurrent Networks

Learning Precise Timing with LSTM Recurrent Networks (2002)

Cached

Download Links

Other Repositories/Bibliography

DBLP

by Felix A. Gers , Nicol N. Schraudolph , Jürgen Schmidthuber

Citations:

27 - 13 self

BibTeX

@MISC{Gers02learningprecise,
    author = {Felix A. Gers and Nicol N. Schraudolph and Jürgen Schmidthuber},
    title = {Learning Precise Timing with LSTM Recurrent Networks},
    year = {2002}
}

Years of Citing Articles

Bookmark

OpenURL

Abstract

The temporal distance between events conveys information essential for numerous sequential tasks such as motor control and rhythm detection. While Hidden Markov Models tend to ignore this information, recurrent neural networks (RNNs) can in principle learn to make use of it. We focus on Long Short-Term Memory (LSTM) because it has been shown to outperform other RNNs on tasks involving long time lags. We find that LSTM augmented by "peephole connections" from its internal cells to its multiplicative gates can learn the fine distinction between sequences of spikes spaced either 50 or 49 time steps apart without the help of any short training exemplars. Without external resets or teacher forcing, our LSTM variant also learns to generate stable streams of precisely timed spikes and other highly nonlinear periodic patterns. This makes LSTM a promising approach for tasks that require the accurate measurement or generation of time intervals.

Citations

1146	The magical number seven, plus or minus two: some limits on our capacity for processing information - Miller - 1956
368	A Learning Algorithm for Continually Running Fully Recurrent Neural Networks’, Neural computation - Williams, Zipser - 1989
214	Learning long-term dependencies with gradient descent is difficult - Bengio, Simard, et al. - 1994
178	Long short-term memory - Hochreiter, Schmidhuber - 1997
119	Gradient calculations for dynamic recurrent neural networks: A survey - Pearlmutter - 1995
105	An efficient gradient-based algorithm for on-line training of recurrent network trajectories - Williams, Peng - 1990
97	An Input Output HMM Architecture - Bengio, Frasconi - 1995
93	Efficient BackProp - LeCun, Bottou, et al. - 1998
76	The utility driven dynamic error propagation network - Robinson, Fallside - 1987
73	Experiments on Learning by BackPropagation - Plaut, Nowlan, et al. - 1986
54	LSTM recurrent networks learn simple context free and context sensitive languages - Gers, Schmidhuber - 2001
51	Learning to Forget: Continual Prediction with LSTM - Gers - 1999
48	Turing computability with neural nets - Siegelmann, Sontag - 1991
33	Gradient flow in recurrent nets: The difficulty of learning long-term dependencies - Hochreiter, Bengio, et al. - 2001
31	A fixed size storage O(n ) time complexity learning algorithm for fully recurrent continually running networks - Schmidhuber - 1992
30	Untersuchungen zu dynamischen neuronalen Netzen. Diploma thesis - Hochreiter - 1991
25	Adaptive neural oscillator using continuous-time backpropagation learning - Doya, Shuji - 1989
21	Learning the initial state of a second-order recurrent neural network during regular-language inference - Forcada, Carrasco - 1995
18	Long short-term memory. Neural Computation, 9, 8, 1735-1780. Primitives and Sequences 61 Ijspeert - Hochreiter, Schmidhuber - 1997
14	The Neural Bucket Brigade: A local learning algorithm for dynamic feedforward and recurrent networks - Schmidhuber - 1989
10	Learning the Long-Term Structure of the Blues - Eck, Schmidhuber
7	pretzel-Wolters, “Existences and learning of oscillations in recurrent neural networks - Townley, WeiB, et al. - 2000
5	Language identification from prosody without explicit features - Cummins, Gers, et al. - 1999
4	Language identi from prosody without explicit features - Cummins, Gers - 1999
3	Gradient in recurrent nets: the diculty of learning long-term dependencies - Hochreiter, Bengio, et al. - 2001
2	Phase-space learning - Cottrell - 1995
2	A recurrent network for modelling noisy temporal sequences. Neurocomputing - Bulsari, Saxén - 1995
1	Learning oscillations using adaptive control - Weiss - 1999
1	Cummins,Felix Gers,and Jürgen Schmidhuber. Language identification from prosody without explicit features - Fred