Abstract

Maximum entropy models are considered by many to be one of the most promising avenues of language modeling research. Unfortunately, long training times make maximum entropy research difficult. We present a novel speedup technique: we change the form of the model to use classes. Our speedup works by creating two maximum entropy models, the first of which predicts the class of each word, and the second of which predicts the word itself. This factoring of the model leads to fewer nonzero indicator functions, and faster normalization, achieving speedups of up to a factor of 35 over one of the best previous techniques. It also results in typically slightly lower perplexities. The same trick can be used to speed training of other machine learning techniques, e.g. neural networks, applied to any problem with a large number of outputs, such as language modeling.

Citations

540	Class-based n-gram models of natural language - Brown, Pietra, et al. - 1992
464	Inducing features of random fields - Pietra, S, et al. - 1997
356	Generalized Iterative Scaling for Log-Linear Models - Darroch, Ratcliff - 1972
154	Adaptive Statistical Language Modeling: A Maximum Entropy Approach - Rosenfeld - 1994
20	Cluster expansions and iterative scaling for maximum entropy language models - Laerty, Suhm - 1996
19	Language model size reduction by pruning and clustering - Goodman, Gao - 2000
2	Efficient Training Methods for Maximum Entropy Language Modeling” ICSLP-2000 - Wun, Khudanpur - 2000
1	Putting it all Together - Goodman - 2000