Learning Long-Term Dependencies with Gradient Descent is Difficult

Learning Long-Term Dependencies with Gradient Descent is Difficult

Cached

Download Links

by Yoshua Bengio , Patrice Simard , Paolo Frasconi

Venue:	TO APPEAR IN THE SPECIAL ISSUE ON RECURRENT NETWORKS OF THE IEEE TRANSACTIONS ON NEURAL NETWORKS
Citations:	214 - 22 self

BibTeX

@MISC{Bengio_learninglong-term,
    author = {Yoshua Bengio and Patrice Simard and Paolo Frasconi},
    title = {Learning Long-Term Dependencies with Gradient Descent is Difficult},
    year = {}
}

Years of Citing Articles

Bookmark

OpenURL

Abstract

Recurrent neural networks can be used to map input sequences to output sequences, such as for recognition, production or prediction problems. However, practical difficulties have been reported in training recurrent neural networks to perform tasks in which the temporal contingencies present in the input/output sequences span long intervals. We showwhy gradient based learning algorithms face an increasingly difficult problem as the duration of the dependencies to be captured increases. These results expose a trade-off between efficient learning by gradient descent and latching on information for long periods. Based on an understanding of this problem, alternatives to standard gradient descent are considered.

Citations

2801	Optimization by simulated annealing - Kirkpatrick, Gelatt, et al. - 1983
2304	Learning Internal Representations by Error Propagation - Rumelhart, Hinton, et al. - 1986
368	A Learning Algorithm for Continually Running Fully Recurrent Neural Networks’, Neural computation - Williams, Zipser - 1989
135	Minimizing Multimodal Functions of Continuous Variables with the ‘Simulated Annealing Algorithm - Corana, Marchesi, et al. - 1987
83	Improving the convergence of back-propagation learning with second order methods - Becker, LeCun - 1989
66	A focused back-propagation algorithm for temporal pattern recognition - Mozer - 1989
63	Global optimization of a neural network—Hidden Markov model hybrid - Bengio, Mori, et al. - 1991
54	Induction of multiscale temporal structure - Mozer - 1992
39	Local feedback multilayered networks - Frasconi, Gori, et al. - 1992
33	Unified integration of explicit rules and learning by example in recurrent networks - Frasconi, Gori, et al. - 1995
25	Learning processes in an asymmetric threshold network, in Disordered systems and biological organisations, Les Houches - LeCun - 1986
22	Inserting Rules into Recurrent Neural Networks - Giles, Omlin
19	BPS: A learning algorithm for capturing the dynamic nature of speech - Gori, Bengio, et al. - 1989
18	The problem of learning long-term dependencies in recurrent networks", invited paper at the - Bengio, Frasconi, et al. - 1993
16	The development of the time-delay neural network architecture for speech recognition - Lang, Hinton - 1988
11	A first look at phonetic discrimination using connectionist models with recurrent links - Kuhn - 1987
11	The moving targets training algorithm - Rohwer - 1989
10	Artificial neural networks and their application to sequence recognition - Bengio - 1991
8	learning by choice of internal representation - Grossman, Meir, et al. - 1988
6	Nonlinear dynamics and stability of analog neural networks - Marcus, Waugh, et al. - 1991
4	Using Random Weights to train Multilayer Networks of Hard-Limiting Units - Bartlett, Downs - 1992
3	Rheinboldt W.C. Iterative Solution of Non-linear Equations - Ortega - 1960
2	Arti cial Neural Networks and their Application to Sequence Recognition - Bengio - 1991
2	Uni ed Integration of Explicit Rules and Learning by Example in Recurrent Networks - Frasconi, Gori, et al. - 1994
1	A Method of Training Multi-layer Networks with Heaviside Characteristics Using Internal Representations - Gaynier, Downs - 1993
1	Learning bychoice of internal representation - Grossman, Meir, et al.