Learning Long-Term Dependencies with Gradient Descent is Difficult

DMCA

Learning Long-Term Dependencies with Gradient Descent is Difficult

Cached

Download Links

by Yoshua Bengio , Patrice Simard , Paolo Frasconi

Venue:	TO APPEAR IN THE SPECIAL ISSUE ON RECURRENT NETWORKS OF THE IEEE TRANSACTIONS ON NEURAL NETWORKS
Citations:	387 - 37 self

BibTeX

@MISC{Bengio_learninglong-term,
    author = {Yoshua Bengio and Patrice Simard and Paolo Frasconi},
    title = {Learning Long-Term Dependencies with Gradient Descent is Difficult},
    year = {}
}

OpenURL

Abstract

Recurrent neural networks can be used to map input sequences to output sequences, such as for recognition, production or prediction problems. However, practical difficulties have been reported in training recurrent neural networks to perform tasks in which the temporal contingencies present in the input/output sequences span long intervals. We showwhy gradient based learning algorithms face an increasingly difficult problem as the duration of the dependencies to be captured increases. These results expose a trade-off between efficient learning by gradient descent and latching on information for long periods. Based on an understanding of this problem, alternatives to standard gradient descent are considered.

Keyphrases

gradient descent long-term dependency prediction problem practical difficulty long period input output sequence algorithm face output sequence input sequence long interval training recurrent neural network temporal contingency recurrent neural network difficult problem

DMCA