Quantitative Evaluation of User Simulation Techniques for Spoken Dialogue Systems (2005)
| Venue: | IN PROC. OF 6TH SIGDIAL |
| Citations: | 30 - 6 self |
BibTeX
@INPROCEEDINGS{Schatzmann05quantitativeevaluation,
author = {Jost Schatzmann and Kallirroi Georgila and Steve Young},
title = {Quantitative Evaluation of User Simulation Techniques for Spoken Dialogue Systems},
booktitle = {IN PROC. OF 6TH SIGDIAL},
year = {2005},
pages = {45--54},
publisher = {}
}
Years of Citing Articles
Bookmark
 |
 |
 |
 |
 |
 |
OpenURL
Abstract
The lack of suitable training and testing data is currently a major roadblock in applying machine-learning techniques to dialogue management. Stochastic modelling of real users has been suggested as a solution to this problem, but to date few of the proposed models have been quantitatively evaluated on real data. Indeed, there are no established criteria for such an evaluation. This paper presents a systematic approach to testing user simulations and assesses the most prominent domain-independent techniques using a large DARPA Communicator corpus of human-computer dialogues. We show that while recent advances have led to significant improvements in simulation quality, simple statistical metrics are still sufficient to discern synthetic from real dialogues. 1
Citations
| 122 | A stochastic model of human-machine interaction for learning dialog strategies - Levin, Pieraccini - 2000 |
| 40 | User Modeling for Spoken Dialogue System Evaluation - Eckert, Levin, et al. - 1997 |
| 24 | Corpus-based dialogue simulation for automatic strategy learning and evaluation - Scheffler, Young - 2001 |
| 18 | Automatic annotation of COMMUNICATOR dialogue data for learning dialogue strategies and user simulations - Georgila, Lemon, et al. - 2005 |
| 13 | Automatic Design of Spoken Dialogue Systems - Scheffler - 2002 |
| 8 | Assessment of dialogue systems by means of a new simulationtechnique - López-Cózar - 2003 |
| 5 | Evaluating dialogue strategies under communication errors using computer-to-computer simulation - Watanabe, Araki, et al. - 1998 |
| 3 | Learning user simulations for information state update dialogue systems - 2005a |
