Abstract

We propose a new hierarchical Bayesian n-gram model of natural languages. Our model makes use of a generalization of the commonly used Dirichlet distributions called Pitman-Yor processes which produce power-law distributions more closely resembling those in natural languages. We show that an approximation to the hierarchical Pitman-Yor language model recovers the exact formulation of interpolated Kneser-Ney, one of the best smoothing methods for n-gram language models. Experiments verify that our model gives cross entropy results superior to interpolated Kneser-Ney and comparable to modified Kneser-Ney. 1

Citations

887	Bayesian Data Analysis - Gelman, Carlin, et al. - 2004
631	J: An Empirical Study of Smoothing Techniques for Language Modeling - SF, Goodman - 1996
449	Probabilistic inference using markov chain monte carlo methods - Neal - 1971
328	Hierarchical Dirichlet processes - Teh, Jordan, et al. - 2006
179	Improved backing-off for m-gram language modeling - Kneser, Ney - 1995
162	The two-parameter PoissonDirichlet distribution derived from a stable subordinator. Annals of Probability - Pitman, Yor - 1997
160	Gibbs sampling methods for stick–breaking priors - Ishwaran, James - 2001
119	Two decades of statistical language modeling: Where do we go from here - Rosenfeld
70	A bit of progress in language modeling - Goodman - 2001
56	Interpolating between types and tokens by estimating power-law generators - Goldwater, Griffiths, et al. - 2006
33	Estimation of probabilities in the language model of the IBM speech recognition system - Nadas - 1984
18	A hierarchical Dirichlet language model. Natural language engineering - MacKay, Peto - 1995
9	Dirichlet processes, Chinese restaurant processes and all that. Tutorial presented at NIPS conference - Jordan - 2005
3	Nonparametric Bayesian methods. Tutorial presented at UAI conference - Ghahramani - 2005