Automatic speech dialogue systems are becoming common. In order to assess their performance, a large sample of real dialogues has to be collected and evaluated. This process is expensive, labor intensive, and prone to errors. To alleviate this situation we propose a user simulation to conduct dialogues with the system under investigation. Using stochastic modeling of real users we can both debug and evaluate a speech dialogue system while it is still in the lab, thus substantially reducing the amount of field testing with real users. 1 Introduction Recent literature shows an increasing number of speech dialogue systems being implemented and used in the field [1, 2, 3, 4, 5, 6]. However, the development of such dialogue systems (especially the dialogue manager) is still considered art rather than an engineering task. While the optimality criteria of a low level component like the speech recognizer are obvious (i.e. generate an accurate transliteration of a speech signal) there exists n...

In this paper we present technology used in spoken dialog systems for applications of a wide range. They include tasks from the travel domain and automatic switchboards as well as large scale directory assistance. The overall goal in developing spoken dialog systems is to allow for a natural and flexible dialog flow similar to human--human interaction. This imposes the challenging task to recognize and interpret user input, where he/she is allowed to choose from an unrestricted vocabulary and an infinite set of possible formulations. We therefore put emphasis on strategies that make the system more robust while still maintaining a high level of naturalness and flexibility. In view of this paradigm, we found that two fundamental principles characterize many of the proposed methods: 1) to consider available sources of information as early as possible, and 2) to keep alternative hypotheses and delay the decision for a single option as long as possible. We describe

In this paper, we present our research on dialog dependent language modeling. In accordance with a speech (or sentence) production model in a discourse we split language modeling into two components; namely, dialog dependent concept modeling and syntactic modeling. The concept model is conditioned on the last question prompted by the dialog system and it is structured using #-grams. The syntactic model , which consists of a collection of stochastic context-free grammars one for each concept, describes word sequences that may be used to express the concepts. The resulting LM is evaluated by rescoring #-best lists. We report significant perplexity improvement with moderate word error rate drop within the context of CU Communicator System; a dialog system for making travel plans by accessing information about flights, hotels and car rentals.

This paper describes a design and feasibility study for a large-scale automatic directory information system with a scalable architecture. The current demonstrator, called PADIS-XL 1, operates in realtime and handles a database of a medium-size German city with 130,000 listings. The system uses a new technique of taking a combined decision on the joint probability over multiple dialogue turns, and a dialogue strategy that strives to restrict the search space more and more with every dialogue turn. During the course of the dialogue, the last name of the desired subscriber must be spelled out. The spelling recognizer permits continuous spelling and uses a context-free grammar to parse common spelling expressions. This paper describes the system architecture, our maximum a-posteriori (MAP) decision rule, the spelling grammar, and the dialogue strategy. We give results on the SPEECHDAT and SIETILL databases on recognition of first names by spelling and on jointly deciding on the spelled and the spoken name. In a 35,000-names setup, the joint decision reduced name-recognition errors by 31%. 1.

The use of Confidence Measures (CMs) in Spoken Dialog System (SDS) applications to suppress the number of verification turns for `reliably correctly recognised utterances' can greatly reduce average dialog length which enhances usability and increases user satisfaction [1]. This paper gives a brief but clear review of the method of CM assessment, which was presented in [2]. It proceeds by demonstrating how the Dutch ARISE (Automatic Railways Information Systems in Europe) SDS was equipped with this technology and shows in deep detail how the parameters involved are to be optimised. The evaluation reveals and explains a typical behaviour of this method with train timetable information-alike systems. This results in a set of conclusions that were not foreseen when the method was first developed for a directory information system. The paper ends with an outlook for solutions in new research directions. 1. INTRODUCTION A number of telephone based travel information systems has been built ...

In the course of a "man-machine" dialogue, the system's belief concerning the user's intention is continuously being built up. Moreover, restricting the discourse to a narrow application domain further constrains the variety of possible user reactions. In this paper, we will show how these knowledge sources may be utilized in a stochastic framework to improve speech understanding. On field-test data collected with our automatic exchange board prototype PADIS, a relative reduction of attribute errors by 27% has been obtained.

In recent years, aquestiT of greatieatTV: has been the development of tools and techni8T# tofaci))T#Z the evaluatiT ofdi:ZG9T systems. The latter can be evaluated fromvari(: poi( ofviZK such asrecogni#ZG and understandi # rates,dis,TVV naturalness and robustnessagaist recognissT errors.EvaluatiZ usually requiyT compim -T a large corpus of words and sentences uttered by users, relevant to theappli:VT#Z domai the systemi desimT9for.Thi paper proposes a newtechni9B that makesi possi(9 to reuse such a corpus for theevaluati# and to check the performance of the system whendinTV)G dinTV)G strategiT are used. ThetechniKZ i based on theautomati generatiT of conversati)) between thediT(B(K system, togetherwie anaddiK9T#( didiK9 system user#si8GG8T#()9 wi8 thediT(GZ: system. Thetechni8G has beenappliV to evaluate a di9:K8: system developedi our labusiV twodiT((ZK recogniT#( front-ends and twodiTZ8:( diTZ8:( strategi# to handle user confirmati(KZ The experiVT#( show that the prompt-dependentrecogniepe front-endachi-en better results, but that thi front-endi appropriVG onlyi users lirs thei utterances to those related to the current system prompt. The prompt-i(9VBKTiK front-endachi-en ihi-en results, but enables front-end users to utter anypermi89G utterance at anytiVB iVB9K(T#(ZB of the system prompt. In consequence,thi front-end may allow a more natural and comfortable imfortableT TheexperiBT#( also show that there-promptiV confirmati strategy enhances system performance for both recogniVT# front-ends.

With the rapid explosion of the World Wide Web, it is becoming increasingly possible to easily acquire a wide variety of information such as flight schedules, yellow pages, used car prices, current stock prices, entertainment event schedules, account balances, etc. It would be very useful to have spoken dialogue interfaces for such information access tasks. We identify portability, usability, robustness, and extensibility as the four primary design objectives for such systems. In other words, the objective is to develop a PURE (Portable, Usable, Robust, Extensible) system.

Abstract not found

The robust estimation of language models for new applications of spoken dialogue systems often suffers from a shortcoming of training material. An alternative to training a language model is to improve an initial language model using material obtained while running the new system, thus adapting it to the new task. In this paper