Abstract

A criterion for pruning parameters from N-gram backoff language models is developed, based on the relative entropy between the original and the pruned model. It is shown that the relative entropy resulting from pruning a single N-gram can be computed exactly and efficiently for backoff models. The relative entropy measure can be expressed as a relative change in training set perplexity. This leads to a simple pruning criterion whereby all N-grams that change perplexity by less than a threshold are removed from the model. Experiments show that a production-quality Hub4 LM can be reduced to 26% its original size without increasing recognition error. We also compare the approach to a heuristic pruning criterion by Seymore and Rosenfeld [9], and show that their approach can be interpreted as an approximation to the relative entropy criterion. Experimentally, both approaches select similar sets of N-grams (about 85% overlap), with the exact relative entropy criterion giving marginally bette...

Citations

6517	Elements of Information Theory - Cover, Thomas - 1991
574	Text Compression - Bell, Cleary, et al. - 1990
574	Estimation of probabilities from sparse data for the language model component of a speech recogniser - Katz - 1987
286	The population frequencies of species and the estimation of population parameters - Good - 1953
124	Hidden Markov Model Induction by Bayesian Model Merging - Stolcke, Omohundro - 1993
69	Tishby N: The Power of Amnesia - Ron, Singer
51	Scalable backoff language models - Seymore, Rosenfeld - 1996
28	Up from trigrams! The struggle for improved language models - Jelinek - 1991
18	Statistical language modeling using a variable context length - Kneser - 1996
5	Acoustic modeling for the SRI Hub4 partitioned evaluation continuous speech recognition system - Sankar, Heck, et al. - 1997
2	Markov model induction by bayesian model merging - Hidden - 1992
1	Scalable backoff languagemodels - Rosenfeld - 1996