DMCA
KL-divergence regularized deep neural network adaptation for improved large vocabulary speech recognition,” ICASSP (2013)
| Citations: | 26 - 3 self |
BibTeX
@MISC{Yu13kl-divergenceregularized,
author = {Dong Yu and Kaisheng Yao and Hang Su and Gang Li and Frank Seide},
title = {KL-divergence regularized deep neural network adaptation for improved large vocabulary speech recognition,” ICASSP},
year = {2013}
}
Share
 |
 |
 |
 |
||
OpenURL
Abstract
We propose a novel regularized adaptation technique for context dependent deep neural network hidden Markov models (CD-DNN-HMMs). The CD-DNN-HMM has a large output layer and many large hidden layers, each with thousands of neurons. The huge number of parameters in the CD-DNN-HMM makes adaptation a challenging task, esp. when the adaptation set is small. The technique developed in this paper adapts the model conservatively by forcing the senone distribution estimated from the adapted model to be close to that from the unadapted model. This constraint is realized by adding Kullback–Leibler divergence (KLD) regularization to the adaptation criterion. We show that applying this regularization is equivalent to changing the target distribution in the conventional backpropagation algorithm. Experiments on Xbox voice search, short message dictation, and Switchboard and lecture speech transcription tasks demonstrate that the proposed adaptation technique can provide 2%-30% relative error reduction against the already very strong speaker independent CD-DNN-HMM systems using different adaptation sets under both supervised and unsupervised adaptation setups. Index Terms — deep neural network, CD-DNN-HMM, speaker adaptation, Kullback–Leibler divergence regularization 1.
Keyphrases
deep neural network adaptation improved large vocabulary speech recognition adaptation set target distribution novel regularized adaptation technique neural network xbox voice search large output layer kullback leibler divergence regularization adaptation criterion lecture speech transcription task index term adaptation technique speaker adaptation different adaptation set short message dictation strong speaker independent cd-dnn-hmm system adapted model many large hidden layer huge number unsupervised adaptation setup senone distribution challenging task relative error reduction kullback leibler divergence unadapted model conventional backpropagation algorithm
