BibTeX
@MISC{Lyu_revisitlong,
author = {Qi Lyu and Jun Zhu},
title = {Revisit Long Short-Term Memory: An Optimization Perspective},
year = {}
}
Share
 |
 |
 |
 |
||
OpenURL
Abstract
Long Short-Term Memory (LSTM) is a deep recurrent neural network archi-tecture with high computational complexity. Contrary to the standard practice to train LSTM online with stochastic gradient descent (SGD) methods, we pro-pose a matrix-based batch learning method for LSTM with full Backpropagation Through Time (BPTT). We further solve the state drifting issues as well as im-proving the overall performance for LSTM using revised activation functions for gates. With these changes, advanced optimization algorithms are applied to LSTM with long time dependency for the first time and show great advantages over SGD methods. We further demonstrate that large-scale LSTM training can be greatly accelerated with parallel computation architectures like CUDA and MapReduce. 1
Keyphrases
revisit long short-term memory optimization perspective parallel computation architecture high computational complexity standard practice full backpropagation stochastic gradient descent great advantage long time dependency deep recurrent neural network archi-tecture activation function short-term memory advanced optimization algorithm overall performance lstm online first time large-scale lstm training matrix-based batch learning method sgd method
