There are now many computer programs for automatically determining the sense in which a word is being used. One would like to be able to say which are better, which worse, and also which words, or varieties of language, present particular problems to which algorithms. An evaluation exercise is required, and such an exercise requires a `gold standard' dataset of correct answers. Producing this proves to be a difficult and challenging task. In this paper I discuss the background, challenges and strategies, and present a detailed methodology for ensuring that the gold standard is not fool's gold. 1 Introduction There are now many computer programs for automatically determining the sense in which a word is being used. One would like to be able to say which are better, which worse, and also which words, or varieties of language, present particular problems to which algorithms. An evaluation exercise is required. A pilot (`SENSEVAL') is taking place under the auspices of ACL SIGLEX (the Le...

This paper describes the development of fast (less than 10 times real-time) large vocabulary continuous speech recognition (LVCSR) systems based on technology developed for unlimited runtime systems assembled for participation in recent DARPA/NIST LVCSR evaluations. A general system structure for 10 times real-time systems is proposed and two specific systems that have been built for Broadcast News (BN) and Conversational Telephone Speech (CTS) recognition are described. The systems were evaluated in the DARPA/NIST April 2003 Rich Transcription evaluation. Results are reported and contrasted with unlimited runtime systems and previous fast systems.

Automatic continuous speech recognition (CSR) is sufficiently