Abstract

In this paper we describe a recent experiment designed to evaluate the performance of the TRAINS-95 system. The evaluation uses a task-based evaluation methodology appropriate for dialogue systems such as TRAINS-95, where a human and a computer interact and collaborate to solve a given problem. In task-based evaluations, techniques are measured in terms of their affect on task performance measures such as how long it takes to develop a solution using the system, and the quality of the final plan produced. The evaluation explores the robustness of the TRAINS-95 system in the presence of word recognition errors, the amount of training required to effectively use the system, and user preferences. This work was supported in part by ONR/ARPA grants N00014-92-J-1512 and N00014-95-1-1088, and NSF grant IRI-9503312. 1 Introduction TRAINS-95 is the first end-to-end implementation in a long-term effort to develop an intelligent planning assistant that is conversationally proficient in natural...

Citations