Abstract

A comparison of algorithms for maximum entropy parameter estimation Conditional maximum entropy (ME) models provide a general purpose machine learning technique which has been successfully applied to fields as diverse as computer vision and econometrics, and which is used for a wide variety of classification problems in natural language processing. However, the flexibility of ME models is not without cost. While parameter estimation for ME models is conceptually straightforward, in practice ME models for typical natural language tasks are very large, and may well contain many thousands of free parameters. In this paper, we consider a number of algorithms for estimating the parameters of ME models, including iterative scaling, gradient ascent, conjugate gradient, and variable metric methods. Surprisingly, the standardly used iterative scaling algorithms perform quite poorly in comparison to the others, and for all of the test problems, a limited-memory variable metric algorithm outperformed the other choices.

Citations

1548	BConditional random fields: Probabilistic models for segmenting and labeling sequence data - Lafferty, McCallum, et al.
1243	Numerical Optimization - Nocedal, Wright - 2006
846	A Maximum Entropy Approach to Natural Language Processing - Berger, Pietra, et al. - 1996
464	Inducing Features of Random Fields - Pietra, Pietra, et al. - 1997
355	Generalized Iterative Scaling for Log-Linear Models - Darroch, Ratcliff - 1972
167	Maximum Entropy Models for natural language ambiguity resolution - Ratnaparkhi - 1993
138	Benchmarking optimization software with performance profiles - Dolan, Moré
125	Estimators for Stochastic “Unification-Based” Grammars - Johnson - 1999
121	Efficient management of parallelism in object oriented numerical software libraries - Balay, Gropp, et al. - 1997
119	Stochastic attribute-value grammars - Abney - 1997
101	PETSc users manual - Balay, Buschelman, et al. - 2009
100	On a least squares adjustment of a sample frequency table when the expected marginal totals are known - Deming, Stephan - 1940
45	Algorithms for maximum-likelihood logistic regression - Minka - 2001
20	Cluster expansions and iterative scaling for maximum entropy language models - Laerty, Suhm - 1996
19	PETSc home - Balay, Buschelman, et al. - 2007
17	Using Maximum Entropy for Sentence Extraction - Osborne - 2002
16	Estimators for stochastic "unification-based" grammars - Johnson, Geman, et al. - 1999
11	Probability Models for Complex Systems - Chi - 1998
5	2001. A limited memory variable metric method in subspaces and bound constrained optimization problems - Benson, More
4	Large scale unconstrained optimization - Nocedal - 1997
2	TAO users manual - Moré, Sarich - 2002
1	PETSc home page. hELp://www.mcs. anl. gov/petsc - Balay, Buschelman, et al. - 2001