Abstract

An experiment with recent test suite and grammar (engineering) resources is outlined: a critical assessment of the EU-funded tsnlp (Test Suites for Natural Language Processing) package as a diagnostic and benchmarking facility for a distributed (multi-site) large-scale hpsg grammar engineering e#ort. This paper argues for a generalized, systematic, and fully automated testing and diagnosis facility as an integral part of the linguistic engineering cycle and gives a practical assessment of existing resources; both a flexible methodology and tools for competence and performance profiling are presented. By comparison to earlier evaluation work as reflected in the Hewlett-Packard test suite data, released exactly ten years before tsnlp, it is judged where testsuite -based evaluation has improved (and where not) over time. 1 Motivation [...] the study and optimisation of unification-based parsing must rely on empirical data until complexity theory can more accurately p...

Citations