Abstract

Unsupervised learning of linguistic structure is a difficult problem. A common approach is to define a generative model and maximize the probability of the hidden structure given the observed data. Typically, this is done using maximum-likelihood estimation (MLE) of the model parameters. We show using part-of-speech tagging that a fully Bayesian approach can greatly improve performance. Rather than estimating a single set of parameters, the Bayesian approach integrates over all possible parameter values. This difference ensures that the learned structure will have high probability over a range of possible parameters, and permits the use of priors favoring the sparse distributions that are typical of natural language. Our model has the structure of a standard trigram HMM, yet its accuracy is closer to that of a state-of-the-art discriminative model (Smith and Eisner, 2005), up to 14 percentage points better than MLE. We find improvements both when training from data alone, and using a tagging dictionary. 1

Citations

3011	Stochastic Relaxation, Gibbs Distributions and the Bayesian Restoration of Images - Geman, Geman - 1984
540	Class-based n-gram models of natural language - Brown, Pietra, et al. - 1992
212	Tagging English text with probabilistic model - Merialdo - 1994
101	Unsupervised Learning of Disambiguation Rules for Part of Speech - Brill - 1995
89	Contrastive Estimation: Training Log-Linear Models on Unlabeled Data - Smith, Eisner - 2005
75	Distributional Part-of-Speech Tagging - Schütze - 1994
72	A generative constituent-context model for improved grammar induction. ACL - Klein, Manning - 2002
66	A hierarchical Dirichlet language model - MacKay, Peto - 1994
63	Aggregate and mixed-order Markov models for statistical language processing - Saul, Pereira - 1997
58	Prototype-driven learning for sequence models - Haghighi, Klein - 2006
46	Inducing syntactic categories by context distribution clustering - Clark - 2000
28	Bayesian inference for PCFGs via Markov chain Monte Carlo - Johnson, Griffiths, et al. - 2007
14	Acquiring syntactic information from distributional statistics. Connectionist models of memory and language - Finch, Chater, et al. - 1995
3	2004. A study of unsupervised partof-speech tagging - Banko, Moore