The GRACE evaluation program aims at applying the Evaluation Paradigm to the evaluation of Part-of-Speech taggers for French. An interesting by-product of GRACE is the production of validated language resources necessary for the evaluation. After a brief recall of the origins and the nature of the Evaluation Paradigm, we show how it relates to other national and international initiatives. We then present the now ending GRACE evaluation campaign and describe its four main components (corpus building, tagging procedure, lexicon building, evaluation procedure), as well as its internal organization. 1. The Evaluation Paradigm The Evaluation Paradigm has been proposed as a mean to foster development in research and technology in the field of language engineering. Up to now, it has been mostly used in the United States in the framework of the ARPA and NIST projects on automatic processing of spoken and written language. The paradigm is based on a two step process: ffl first, create textual...

This paper presents a comparative evaluation of ready-to-use, XML-based tools for annotating linguistic data. We start by describing our research project that deals with the creation and annotation of empirical data related to information structure. Based on the requirements of this project and the data, we develop a set of evaluation criteria and apply them in the evaluation of five selected annotation tools.

An Evaluation of LOLITA and related Natural Language Processing Systems Paul Callaghan Submitted to the University of Durham for the degree of Ph.D., August 1997 --------------------- This research addresses the question, "how do we evaluate systems like LOLITA?" LOLITA is the Natural Language Processing (NLP) system under development at the University of Durham. It is intended as a platform for building NL applications. We are therefore interested in questions of evaluation for such general NLP systems. The thesis has two parts.

. Although undeniably useful for the translation of certain types of repetitive document, current translation memory technology is limited by the rudimentary techniques employed for approximate matching. Such systems, moreover, incorporate no real notion of a document, since the databases that underlie them are essentially composed of isolated sentence strings. As a result, current TM products can only exploit a small portion of the knowledge residing in translators' past production. This paper examines some of the changes that will have to be implemented if the technology is to be made more widely applicable. 1 Introduction The term "translation memory" admits of at least two different definitions, one broad and one narrow. The narrower, but more widely used, definition corresponds to the characteristics of a popular set of commercial products that includes Translator's Workbench from Trados, Transit from Star AG, D'ej`aVu from Atril and IBM's TranslationManager/2. Accordin...

This paper reports on an evaluation of how well a specific MT system would perform in translating new text-types including an assessment of in what ways the system itself could be extended to deal with new text-types. The overall evaluation and quality criterion is defined in terms of how much effort it takes to post-edit the text after having been translated by the MT system. A structured questionnaire rating different error types was given to the post-editors involved. The results were then "translated " into a number of mainly linguistic phenomena occurring in the input text causing these errors. In order to achieve consistency and reliability the analysis of the new text-types was automated as far as possible. A suite of programs was developed, each of which identifies a particular phenomenon and assigns scores for each occurrence. A reference text, known as being a good text, was first analysed using the procedure in order to provide a benchmark against which to assess the results from analysing the new text-types. After running the evaluation, a representative subset of the new text-types were then selected and translated by a slightly revised version of the MT system and assessed by the post-editors (using the same questionnaire).

Three different ways of evaluating a Swedish grammar checker are presented and discussed in this article. The first evaluation concerns measuring the program's detection capacity on five text genres. The measures (precision and recall) are often used in evaluating grammar checkers. However, in order to test and improve the usability of grammar checking software, they need to be complemented with user-oriented methods. Consequently, the second and the third evaluations presented in the article both involve users. The second evaluation focuses on user reactions to grammar error presentations, especially with regard to false alarms and erroneous error identification. The third and last evaluation focuses on problems in supporting users ' cognitive revision processes. It also examines user motives behind choosing to correct or not to correct problems highlighted by the program. Advantages and disadvantages of the different evaluation methods are discussed. 1.

The CONCERTO project is concerned with the creation and management of knowledge repositories. The distinctive approach is to maintain an association between the textual form in which knowledge is expressed in source documents, and an expressive narrative knowledge representation language that supports inference and query operations. We first situate the CONCERTO approach in relation to current principles of knowledge management, before exploring three aspects of the mechanisms that CONCERTO supports: document management, acquisition of knowledge from text, and annotation base management. The concluding section gives an insight into how these mechanisms are being translated into changing working practices in a knowledge-based organisation within the CONCERTO consortium. 1 Principles of Knowledge Management In recent years, knowledge management has come to prominence as a topic of major concern for organisations (Davenport & Prusak, 1998). However, it is only very recently that academic discussion has moved towards practical implementation on the large scale,

states of mind, etc. Natural language semantics, as a theory of meaning, determines what the meanings of words of the language are and how to semantically combine elements of a language to build up complex meanings. These meanings are most often represented as formulas in a logical language. Computational semantics investigates the computational properties that formal semantic theories need to enjoy to be applicable to real-world problems. Most approaches in formal semantics are evaluated with respect to criteria like expressiveness, explanatory adequacy, and generality, whereas typical criteria for evaluating approaches in computational semantics include coverage, robustness, eciency, and user-friendliness. Coverage measures the amount of phenomena that can described by a semantic theory. Although coverage should be regarded as a criterion f

During the last few years, word graphs have been gaining increasing interest within the speech community as the primary interface between speech recognizers and language processing modules. Both development and evaluation of graphproducing speech decoders require generally accepted measures of word graph quality. While the notion of recognition accuracy can easily be extended to word graphs, a meaningful measure of word graph size has not yet surfaced. We argue, that the number of derivation steps a theoretical parser would need to process all unique sub-paths in a graph could provide a measure that is both application oriented enough to be meaningful and general enough to allow a useful comparison of word recognizers across different applications. This paper discusses various measures that are used, or could be used, to measure word graph quality. Using real-life data (word graphs evaluated in the 1996 Verbmobil acoustic evaluation), it is demonstrated how different measures can affec...

The ISLE project is a continuation of the long standing EAGLES initiative, carried out under the Human Language Technology (HLT) programme in collaboration between American and European groups in the framework of the EU-US International Research Co-operation, supported by NSF and EC. We concentrate in this paper on the current position of the ISLE Computational Lexicon Working Group. We provide a short description of the EU SIMPLE lexicons built on thebasisofpreviousEAGLES recommendations. We then point at a few basic methodological principles applied in previous EAGLES phases, and describe a few principles to be followed in the definition of a Multilingual ISLE Lexical Entry (MILE).