Abstract

A fundamental problem in statistical parsing is the choice of criteria and algorithms used to estimate the parameters in a model. The predominant approach in computational linguistics has been to use a parametric model with some variant of maximum-likelihood estimation. The assumptions under which maximum-likelihood estimation is justified are arguably quite strong. This paper discusses the statistical theory underlying various parameter-estimation methods, and gives algorithms which depend on alternatives to (smoothed) maximumlikelihood estimation. We first give an overview of results from statistical learning theory. We then show how important concepts from the classification literature -- specifically, generalization results based on margins on training data -- can be derived for parsing models. Finally, we describe parameter estimation algorithms which are motivated by these generalization bounds.

Citations

6696	The Nature of Statistical Learning Theory - Vapnik - 1995
3377	Introduction to Automata Theory, Languages, and Computation - Hopcroft, Motwani, et al. - 2003
1714	Schapire R: A decision-theoretic generalization of online learning and an application to boosting - Freund - 1997
1548	BConditional random fields: Probabilistic models for segmenting and labeling sequence data - Lafferty, McCallum, et al.
1515	A Theory of the Learnable - Valiant - 1984
1491	Support-vector networks - Cortes, Vapnik - 1995
1086	Making large-scale SVM learning practical - Joachims - 1999
1057	An Introduction to Support Vector Machines and Other Kernel-based Learning Methods - Cristianini, Shawe-Taylor - 2000
896	Additive logistic regression: a statistical view of boosting - Friedman, Hastie, et al. - 2000
816	On the uniform convergence of relative frequencies of events to their probabilities. Theory of Prob. and its - Vapnik, Chervonenkis - 1971
801	A probabilistic theory of pattern recognition - Devroye, Györfi, et al. - 1996
780	Head-Driven Statistical Models for Natural Language Parsing - Collins - 1999
758	JC: Fast Training of Support Vector Machines using Sequential Minimal Optimization - Platt - 1998
606	W: Boosting the Margin: A New Explanation for the Effectiveness of Voting Methods. The Annals of Statistics - Schapire, Freund, et al. - 1998
599	The perceptron: A probabilistic model for information storage and organization - Rosenblatt - 1958
464	Inducing Features of Random Fields - Pietra, Pietra, et al. - 1997
383	An efficient boosting algorithm for combining preferences - Freund, Iyer, et al. - 1998
340	BDiscriminative training methods for hidden Markov models - Collins
311	Large margin classification using the perceptron algorithm - Freund, Schapire - 1999
274	A Maximum Entropy Part-Of-Speech Tagger - Ratnaparkhi - 1996
255	PL: Neural Network Learning: Theoretical Foundations - Anthony, Bartlett - 1999
239	On the algorithmic implementation of multiclass kernel-based vector machines - Crammer, Singer - 2001
220	Discriminative Reranking for Natural Language Parsing - Collins - 2000
204	Convolution kernels for natural languages - Collins, Duffy - 2001
171	Y.: Logistic regression, AdaBoost and Bregman distances - Collins, Schapire, et al. - 2002
164	N: New Ranking Algorithms for Parsing and Tagging: Kernels over Discrete Structures, and the Voted Perceptron - Collins, Duffy
156	The sample complexity of pattern classification with neural networks: the size of the weights is more important than the size of the network - Bartlett
129	Beyond Grammar: An Experience-Based Theory of Language. Center for the Study of Language and Inf - Bod - 1998
125	Estimators for Stochastic “Unification-Based” Grammars - Johnson - 1999
119	Stochastic attribute-value grammars - Abney - 1997
97	On convergence proofs on perceptrons - Novikoff - 1962
86	Applying Probability Measures to Abstract Languages - Booth, Thompson - 1973
69	Linear programming boosting via column generation - Demiriz, Bennett, et al.
68	The Perceptron: a model for brain functioning - Block - 1962
66	Boosting and maximum likelihood for exponential models - Lebanon, Lafferty
60	Ranking algorithms for named-entity extraction: Boosting and the voted perceptron - Collins - 2002
50	Relating data compression and learnability - Littlestone, Warmuth - 1986
36	Additive models, boosting, and inference for generalized divergences - Lafferty - 1999
24	Spot: A trainable sentence planner - Walker, Rambow, et al. - 2001
22	A maximum entropy model for parsing - Ratnaparkhi, Roukos, et al. - 1994
16	Estimators for stochastic "unification-based" grammars - Johnson, Geman, et al. - 1999
11	Parsing with a Single Neuron: Convolution Kernels for Natural Language Problems - Collins, Duffy - 2001
9	Margin error and generalization capabilities of multi-class discriminant models - Elissee, Guermeur, et al. - 1999
4	Estimators for stochastic `unification-based" grammars - Johnson, Geman, et al. - 1999
4	Margin error and generalization capabilities of multiclass discriminant systems - Elisseeff, Guermeur, et al. - 1999
1	Minimization over Data-Dependent Hierarchies - Risk - 1998