This report presents a detailed analysis and review of NLP evaluation, in principle and in practice. Part 1 examines evaluation concepts and establishes a framework for NLP system evaluation. This makes use of experience in the related area of information retrieval and the analysis also refers to evaluation in speech processing. Part 2 surveys significant evaluation work done so far, for instance in machine translation, and discusses the particular problems of generic system evaluation. The conclusion is that evaluation strategies and techniques for NLP need much more development, in particular to take proper account of the influence of system tasks and settings. Part 3 develops a general approach to NLP evaluation, aimed at methodologically-sound strategies for test and evaluation motivated by comprehensive performance factor identification. The analysis throughout the report is supported by extensive illustrative examples. This work was carried out under the UK Science and Engineeri...

The Traffic Information Collator (TIC) 1 (Allport, 1988, 1989) is a prototype system which takes verbatim police reports of traffic incidents, interprets them, builds a picture of what is happening on the roads and broadcasts appropriate messages automatically to motorists where necessary. Cahill and Evans (1990) described the process of converting the main TIC lexicon (a lexicon of around 1000 words specific to the domain of traffic reports) into DATR (Evans and Gazdar, 1989a, 1989b, 1990). This paper reviews the strategy adopted in the conversion discussed in that paper, and discusses the results of converting the whole lexicon, together with statistics comparing efficiency and performance between the original lexicon and the DATR version. 1 Introduction The Traffic Information Collator (TIC) is a prototype system which takes verbatim police reports of traffic incidents, interprets them, builds a picture of what is happening on the roads and broadcasts appropriate messages automat...

en by semantic expectations, ignoring intervening text not matching these expectations [1]; this is robust but can lead to serious en'om. Another approach has been to identify "interesting" words and attempt only partial sentence parses around those words [2], As an alter- native, we have explored the use of full syntactic analysis of the input, coupled with preference semantics. Preference semantics, as introduced by Wilks [31, penalizes but. does not reject analyses which violate semantic constraints; it selects the analysis with the lbwest constraint violations. The task to which we have applied preference semantics is that of (;mating a data base from U S Navy messages describing nawtl encounters. 'lqaesc messages are relatively brief (average length 30 words) and are highly telegraphic. with many sentence tkagmcnts and frequent run-on sentences. The specific task was to identify live (;lasses of events within these messages and, lbr each event, identify the initiating force (frie