A Maximum Entropy Approach to Adaptive Statistical Language Modeling

A Maximum Entropy Approach to Adaptive Statistical Language Modeling (1996)

Cached

Download Links

by Ronald Rosenfeld

Venue:	Computer, Speech and Language
Citations:	201 - 11 self

Versions

Version History

Metadata Version 0

Datum	Value	Source
TITLE	A Maximum Entropy Approach to Adaptive Statistical Language Modeling	SVM HeaderParse 0.1
AUTHOR NAME	Ronald Rosenfeld	SVM HeaderParse 0.1
AUTHOR ADDR	Pittsburgh, PA 15213 USA	SVM HeaderParse 0.1
ABSTRACT	An adaptive statistical languagemodel is described, which successfullyintegrates long distancelinguistic information with other knowledge sources. Most existing statistical language models exploit only the immediate history of a text. To extract information from further back in the document's history, we propose and use trigger pairs as the basic information bearing elements. This allows the model to adapt its expectations to the topic of discourse. Next, statistical evidence from multiple sources must be combined. Traditionally, linear interpolation and its variants have been used, but these are shown here to be seriously deficient. Instead, we apply the principle of Maximum Entropy (ME). Each information source gives rise to a set of constraints, to be imposed on the combined estimate. The intersection of these constraints is the set of probability functions which are consistent with all the information sources. The function with the highest entropy within that set is the ME solution. Given consistent statistical evidence, a unique ME solution is guaranteed to exist, and an iterative algorithm exists which is guaranteed to converge to it. The ME framework is extremely general: any phenomenon that can be described in terms of statistics of the text can be readily incorporated. An adaptive language model based on the ME approach was trained on the Wall Street Journal corpus, and showed 32%--39 % perplexity reduction over the baseline. When interfaced to SPHINX-II, Carnegie Mellon's speech recognizer, it reduced its error rate by 10%--14%. This thus illustrates the feasibility of incorporating many diverse knowledge sources in a single, unified statistical framework. 1	SVM HeaderParse 0.1
CITATIONS	49 found	ParsCit 1.0