Abstract

We describe a corpus-based induction algorithm for probabilistic context-free grammars. The algorithm employs a greedy heuristic search within a Bayesian framework, and a post-pass using the Inside-Outside algorithm. We compare the performance of our algorithm to n-gram models and the Inside-Outside algorithm in three language modeling tasks. In two of the tasks, the training data is generated by a probabilistic context-free grammar and in both tasks our algorithm outperforms the other techniques. The third task involves naturally-occurring data, and in this task our algorithm does not perform as well as n-gram models but vastly outperforms the Inside-Outside algorithm. 1

Citations

6231	Maximum likelihood from incomplete data via the EM algorithm - Dempster, Laird, et al. - 1977
1412	An Introduction to Kolmogorov Complexity and Its Applications - Li, Vitanyi - 1997
891	Modeling by shortest data description - Rissanen - 1978
540	Class-based n-gram models of natural language - Brown, Pietra, et al. - 1992
345	A maximum likelihood approach to continuous speech recognition - Bahl, Jelinek - 1983
316	The estimation of stochastic context-free grammars using the inside-outside algorithm. Computer Speech and Language - Lari, Young - 1990
316	A formal theory of inductive inference - Solomonoff - 1964
286	Inductive inference: Theory and methods - ANGLUIN, SMITH - 1983
286	Interpolated estimation of Markov source parameters from sparse data - Jelinek, Mercer - 1980
279	Prediction and entropy of printed English - Shannon - 1951
240	Inside-Outside Reestimation from Partially Bracketed Corpora - Pereira, Schabes - 1992
232	Trainable Grammars for Speech Recognition - Baker - 1979
173	An inequality with applications to statistical estimation for probabilistic functions of a Markov process and to models for ecology - Baum, Eagon - 1967
120	Modeling by shortest data description. Automatica 14 - Rissanen - 1978
112	Stochastic lexicalized tree-adjoining grammars - Schabes - 1992
83	Probabilistic tree-adjoining grammar as a framework for statistical natural language processing - Resnik - 1992
63	The DRAGON system – an overview - Baker - 1975
63	A Spelling Correction Program Based on a Noisy Channel Model - Kernighan, Church, et al. - 1990
38	Applications of stochastic context-free grammars using the inside-outside algorithm - Lari, Young - 1991
31	A preliminary report on a general theory of inductive inference - Solomonoff - 1960
9	Recognition of a continuously read natural corpus - Bahl, Baker, et al. - 1978
7	A maximum likelihood approach to continuous speech recognition - Mercer - 1983
7	Combining statistical and syntactic methods in recognizing handwritten sentences - Srihari, Baltus - 1992
6	Empirical acquisition of word and phrase classes in the ATIS domain - McCandless, Glass - 1993
1	Class-based n-gram models of natural language - Cook, Rosenfeld, et al. - 1992
1	Grammatical inference by hill climbing - Aronson - 1976
1	A preliminary report on a general theory of inductive inference - R - 1960
1	Inductive inference: theory and methods. ACM Computing Surveys, 15:237– 269. et al.1978 - Angluin, Smith - 1983