Abstract

With the emergence of broad-coverage parsers, quantitative evaluation of parsers becomes increasingly more important. We propose a dependency-based method for evaluating broad-coverage parsers. The method offers several advantages over previous methods that are based on phrase boundaries. The error count score we propose here is not only more intuitively meaningful than other scores, but also more relevant to semantic interpretation. We will also present an algorithm for transforming constituency trees into dependency trees so that the evaluation method is applicable to both dependency and constituency grammars. Finally, we discuss a set of operations for modifying dependency trees that can be used to eliminate inconsequential differences among different parse trees and allow us to selectively evaluate different aspects of a parser. 1. Introduction With the emergence of broad-coverage parsers, quantitative evaluation of parsers becomes increasingly more important. It is generally acc...

Citations

296	Dependency Syntax: Theory and Practice - Melcuk - 1988
184	A Procedure for Quantitatively Comparing the Syntactic Coverage of English Grammars - Black - 1991
136	Natural Language Parsing as Statistical Pattern Recognition - Magerman - 1994
50	Development and evaluation of a broad-coverage probabilistic grammar of English-language computer manuals - Black, Lafferty, et al. - 1992