DMCA
Marginalized Stacked Denoising Autoencoders
| Citations: | 3 - 1 self |
BibTeX
@MISC{Chen_marginalizedstacked,
author = {Minmin Chen and Zhixiang (eddie Xu and Kilian Q. Weinberger and Fei Sha},
title = {Marginalized Stacked Denoising Autoencoders},
year = {}
}
Share
 |
 |
 |
 |
||
OpenURL
Abstract
Stacked Denoising Autoencoders (SDAs) [4] have been used successfully in many learning scenarios and application domains. In short, denoising autoencoders (DAs) train one-layer neural networks to reconstruct input data from partial random corruption. The denoisers are then stacked into deep learning architectures where the weights are fine-tuned with back-propagation. Alternatively, the outputs of intermediate layers can be used as input features to other learning algorithms. These learned feature representations are known to improve classification accuracies in many cases. For example, Glorot et. al. [3] applied SDAs to domain adaptation and demonstrated that these learned features, when used with a simple linear SVM classifier, yield record performance in benchmark sentiment analysis tasks [1]. One downside of SDAs is the arguably long training time, which often entails specialized computing supports such as GPUs, especially for large-scale tasks. In this abstract we propose a variation to SDAs, in which the random corruption is marginalized out. This crucial step yields the optimal reconstruction weights computed in closed-form and eliminates the use of back-propagation for tuning. We show that the features learned with our approach lead to comparable classification accuracy as SDAs’. The training time, however, reduces by orders of magnitude – from up to 19 hours for SDAs to mere 3 minutes with our approach. Linear Denoiser. The basic building block of our framework is a one-layer linear denoising autoencoder. From a given set of inputs D, we sample inputs x1,... xm with replacement, where typically m> |D|. We corrupt these
Keyphrases
large-scale task random corruption input data simple linear svm classifier many learning scenario deep learning architecture intermediate layer optimal reconstruction weight many case one-layer neural network learned feature representation yield record performance basic building block input x1 crucial step application domain partial random corruption input feature approach lead long training time training time linear denoiser classification accuracy comparable classification accuracy benchmark sentiment analysis task
