Abstract

We describe a speedup for training conditional maximum entropy models. The algorithm is a simple variation on Generalized Iterative Scaling, but converges roughly an order of magnitude faster, depending on the number of constraints, and the way speed is measured. Rather than attempting to train all model parameters simultaneously, the algorithm trains them sequentially. The algorithm is easy to implement, typically uses only slightly more memory, and will lead to improvements for most maximum entropy problems.

Citations

1548	BConditional random fields: Probabilistic models for segmenting and labeling sequence data - Lafferty, McCallum, et al.
846	A Maximum Entropy Approach to Natural Language Processing - Berger, Pietra, et al. - 1996
612	Statistical Methods for Speech Recognition - Jelinek - 1997
464	Inducing Features of Random Fields - Pietra, Pietra, et al. - 1997
355	Generalized Iterative Scaling for Log-Linear Models - Darroch, Ratcliff - 1972
181	R: A Gaussian prior for smoothing maximum entropy models - SF, Rosenfeld - 1999
171	Y.: Logistic regression, AdaBoost and Bregman distances - Collins, Schapire, et al. - 2002
167	Maximum Entropy Models for natural language ambiguity resolution - Ratnaparkhi - 1993
154	Adaptive Statistical Language Modeling: A Maximum Entropy Approach - Rosenfeld - 1994
145	A maximum entropy approach to identifying sentence boundaries - Reynar, Ratnaparkhi - 1997
8	Mitigating the paucity of data problem - Banko, Brill - 2001
5	Gibbs-markov models - Lafferty - 1995
1	A note on approximations to probability distributions - Brown - 1959