We present a stochastic parsing system consisting of a Lexical-Functional Grammar (LFG), a constraint-based parser and a stochastic disambiguation model. We report on the results of applying this system to parsing the UPenn Wall Street Journal (WSJ) treebank. The model combines full and partial parsing techniques to reach full grammar coverage on unseen data. The treebank annotations are used to provide partially labeled data for discriminative statistical estimation using exponential models. Disambiguation performance is evaluated by measuring matches of predicate-argument relations on two distinct test sets. On a gold standard of manually annotated f-structures for a subset of the WSJ treebank, this evaluation reaches 79% F-score. An evaluation on a gold standard of dependency relations for Brown corpus data achieves 76% F-score.

This paper describes a number of log-linear parsing models for an automatically extracted lexicalized grammar. The models are "full" parsing models in the sense that probabilities are defined for complete parses, rather than for independent events derived by decomposing the parse tree. Discriminative training is used to estimate the models, which requires incorrect parses for each sentence in the training data as well as the correct parse. The lexicalized grammar formalism used is Combinatory Categorial Grammar (CCG), and the grammar is automatically extracted from CCGbank, a CCG version of the Penn Treebank. The combination of discriminative training and an automatically extracted grammar leads to a significant memory requirement (over 20 GB), which is satisfied using a parallel implementation of the BFGS optimisation algorithm running on a Beowulf cluster. Dynamic programming over a packed chart, in combination with the parallel implementation, allows us to solve one of the largest-scale estimation problems in the statistical parsing literature in under three hours. A key component of the parsing system, for both training and testing, is a Maximum Entropy supertagger which assigns CCG lexical categories to words in a sentence. The supertagger makes the discriminative training feasible, and also leads to a highly efficient parser. Surprisingly,

In this paper we discuss the construction, features, and current uses of the PARC 700 DEPBANK. The PARC 700 DEPBANK is a dependency bank containing predicate-argument relations and a wide variety of other grammatical features.

We investigate some pitfalls regarding the discriminatory power of MT evaluation metrics and the accuracy of statistical significance tests. In a discriminative reranking experiment for phrase-based SMT we show that the NIST metric is more sensitive than BLEU or F-score despite their incorporation of aspects of fluency or meaning adequacy into MT evaluation. In an experimental comparison of two statistical significance tests we show that p-values are estimated more conservatively by approximate randomization than by bootstrap tests, thus increasing the likelihood of type-I error for the latter. We point out a pitfall of randomly assessing significance in multiple pairwise comparisons, and conclude with a recommendation to combine NIST with approximate randomization, at more stringent rejection levels than is currently standard.

It is not clear a priori how well parsers trained on the Penn Treebank will parse significantly different corpora without retraining. We carried out a competitive evaluation of three leading treebank parsers on an annotated corpus from the human molecular biology domain, and on an extract from the Penn Treebank for comparison, performing a detailed analysis of the kinds of errors each parser made, along with a quantitative comparison of syntax usage between the two corpora. Our results suggest that these tools are becoming somewhat over-specialised on their training domain at the expense of portability, but also indicate that some of the errors encountered are of doubtful importance for information extraction tasks. Furthermore, our inital experiments with unsupervised parse combination techniques showed that integrating the output of several parsers can ameliorate some of the performance problems they encounter on unfamiliar text, providing accuracy and coverage improvements, and a novel measure of trustworthiness. Supplementary materials are available at

Abstract not found

Data-driven grammatical function tag assignment has been studied for English using the Penn-II Treebank data. In this paper we address the question of whether such methods can be applied successfully to other languages and treebank resources. In addition to tag assignment accuracy and f-scores we also present results of a task-based evaluation. We use three machine-learning methods to assign Cast3LB function tags to sentences parsed with Bikel’s parser trained on the Cast3LB treebank. The best performing method, SVM, achieves an f-score of 86.87 % on gold-standard trees and 66.67 % on parser output- a statistically significant improvement of 6.74 % over the baseline. In a task-based evaluation we generate LFG functional-structures from the functiontag-enriched trees. On this task we achive an f-score of 75.67%, a statistically significant 3.4 % improvement over the baseline. 1

We describe how the British National Corpus (BNC) [4], a 100 million word balanced corpus of British English, was parsed into Lexical Functional Grammar (LFG) [8] c-structures and f-structures using a treebankbased parsing architecture. The parsing architecture uses a state-of-the-art statistical parser trained on the Penn Treebank (PTB) [6, 9] to produce c-structures, and an annotation algorithm [5] to enrich the cstructures with corresponding f-structures. We present some issues encountered in applying the parsing architecture on such a large scale. We describe how 1,000 BNC sentences were manually parsed to produce a c-structure gold standard. We present results obtained by evaluating the c-structures produced by the statistical parser against the c-structure gold standard. Finally, we present results obtained by evaluating the f-structures produced by the annotation algorithm against an automatically constructed f-structure gold standard for the 1,000 hand-parsed BNC sentences. To our knowledge, the BNC has not been parsed in its entirety using a deep linguistic grammar formalism. [1] describe how a broad coverage HPSG grammar of English was applied to BNC sentences – the aim of that work was not to parse the BNC but to test the grammar’s coverage using a small subsection of the BNC. Our research demonstrates that it is feasible to provide a reasonably accurate LFG analysis of a very large body of sentences in a robust, non-labour-intensive way. To facilitate parsing with a PTB-trained parser, some reversible transformations were applied to the BNC

A dissertation submitted in partial fulfilment of the requirements for the award of Doctor of Philosophy to the

to basic predicate-argument-modifier (dependency) structure or simple logical form (van Genabith and Crouch, 1996; Cahill et al., 2003a). A number of methods have been