Abstract

Some models of textual corpora employ text generation methods involving n-gram statistics, while others use latent topic variables inferred using the “bag-of-words ” assumption, in which word order is ignored. Previously, these methods have not been combined. In this work, I explore a hierarchical generative probabilistic model that incorporates both n-gram statistics and latent topic variables by extending a unigram topic model to include properties of a hierarchical Dirichlet bigram language model. The model hyperparameters are inferred using a Gibbs EM algorithm. On two data sets, each of 150 documents, the new model exhibits better predictive accuracy than either a hierarchical Dirichlet bigram language model or a unigram topic model. Additionally, the inferred topics are less dominated by function words than are topics discovered using unigram statistics, potentially making them more meaningful. 1.

Citations

1370	Latent Dirichlet allocation - Blei, Ng, et al.
717	Bayes factors - Kass, Raftery - 1995
372	Finding scientific topics - Griffiths, Steyvers - 2004
286	Interpolated estimation of Markov source parameters from sparse data - Jelinek, Mercer - 1980
89	Integrating topics and syntax - Griffiths, Steyvers, et al. - 2005
66	A hierarchical Dirichlet language model - MacKay, Peto - 1994
7	Estimating a Dirichlet distribution. http://research.microsoft.com/∼minka/papers/dirichlet - Minka - 2000
5	Matlab Topic Modeling Toolbox. http://psiexp.ss.uci.edu/research/programs_data/toolbox.htm - Steyvers, Griffiths