Abstract

This work shows how a dialogue model can be represented as a Partially Observable Markov Decision Process (POMDP) with observations composed of a discrete and continuous component. The continuous component enables the model to directly incorporate a confidence score for automated planning. Using a testbed simulated dialogue management problem, we show how recent optimization techniques are able to find a policy for this continuous POMDP which outperforms a traditional MDP approach. Further, we present a method for automatically improving handcrafted dialogue managers by incorporating POMDP belief state monitoring, including confidence score information. Experiments on the testbed system show significant improvements for several example handcrafted dialogue managers across a range of operating conditions. 1

Citations

3118	A tutorial on hidden Markov models and selected applications in speech recognition - Rabiner - 1989
1135	Learning from delayed rewards - Watkins - 1989
979	How to Do Things With Words - Austin - 1962
629	Planning and acting in partially observable stochastic domains - Kaelbling, Littman, et al. - 1998
500	Assessing agreement on classification tasks: the kappa statistic - Carletta - 1996
498	Markov decision processes - Puterman - 1995
423	Bayesian networks and decision graphs - Jensen - 2001
285	The Optimal Control of Partially Observable Markov Processes - Sondik - 1971
243	Acting optimally in partially observable stochastic domains - Cassandra, Kaelbling, et al. - 1994
235	The optimal control of partially observable Markov processes over a finite horizon - Smallwood, Sondik - 1973
185	Point-based value iteration: An anytime algorithm for POMDPs - Pineau, Gordon, et al.
165	Survey of partially observable markov decision processes: Theory, models, and algorithms - Monahan - 1982
139	A Computational Theory of Grounding in Natural Language Conversation - Traum - 1994
128	A taxonomy of illocutionary acts - Searle - 1975
122	A stochastic model of human-machine interaction for learning dialog strategies - Levin, Pieraccini - 2000
111	Perseus: Randomized point-based value iteration for POMDPs - Spaan, Vlassis
105	PARADISE: a framework for evaluating spoken dialogue agents - Walker, Litman, et al. - 1998
99	Optimal control of Markov decision processes with incomplete state estimation - Astrom - 1965
98	Optimizing Dialogue Management with Reinforcement Learning: Experiments with the NJFun System - Singh, Litman, et al. - 2002
94	Computing optimal policies for partially observable decision processes using compact representations - Boutilier, Poole - 1996
91	Speech and language processing - Jurafsky, Martin - 2000
90	Information state and dialogue management in the TRINDI dialogue move engine toolkit - Larsson, Traum
87	Solving POMDPs by search in policy space - Hansen - 1998
72	Towards developing general models of usability with - Walker, Kamm, et al. - 2000
70	Spoken Natural Language Dialog Systems: A Practical Approach - Smith, R, et al. - 1994
61	Vector-based natural language call routing - Chu-Carroll, Carpenter - 1999
61	Conversational interfaces: Advances and challenges - Zue, Glass - 2000
57	Spoken dialogue management using probabilistic reasoning - Roy, Pineau, et al.
57	Exponential family PCA for belief compression in POMDPs - Roy, Gordon - 2003
52	Estimating confidence using word lattices - Kemp, Schaaf - 1997
51	Characterizing and recognizing spoken corrections in human-computer dialogue - Levow - 1998
50	Hybrid Reinforcement/Supervised Learning of Dialogue Policies from Fixed Data Sets - Henderson, Lemon, et al. - 2008
48	A Stochastic Model of ComputerHuman Inter-Action for Learning Dialog Strategies - Levin, Pieraccini - 1997
47	Coding dialogues with the DAMSL annotation scheme - Core, Allen - 1997
47	Optimal Learning: Computational procedures for Bayes-adaptive Markov decision processes - Duff - 2002
47	Learning optimal dialogue strategies: A case study of a spoken dialogue agent for email - Walker, Fromer, et al. - 1998
46	Dynamic programming for POMDPs using a factored state representation - Hansen, Feng - 2000
43	Using markov decision process for learning dialogue strategies - Levin, Pieraccini - 1998
40	User Modeling for Spoken Dialogue System Evaluation - Eckert, Levin, et al. - 1997
40	Dialogue Management in the MERCURY Flight Reservation System - Seneff, Polifroni - 2000
36	A Framework for Unsupervised Learning of Dialogue Strategies - Pietquin - 2004
36	Conversation as action under uncertainty - Paek, Horvitz - 2000
36	VDCBPI: An approximate scalable algorithm for large scale POMDPs - Poupart, Boutilier - 2004
35	2002. Automatic learning of dialogue strategy using dialogue simulation and reinforcement learning - Scheffler, Young
33	Posterior probability decoding, confidence estimation and system combination - Evermann, Woodland - 2000
32	Factored Partially Observable Markov Decision Processes for Dialogue Management - Williams, Poupart, et al. - 2005
32	Evaluating Spoken Dialogue Agents with {PARADISE}: Two Case Studies. Computer Speech and Language - Walker, Litman, et al. - 1998
31	Observation of a Markov process through a noisy channel - Drake - 1962
31	Combining acoustic and pragmatic features to predict recognition performance in spoken dialogue systems - Gabsdil, Lemon - 2004
30	Quantitative evaluation of user simulation techniques for Spoken Dialogue Systems - Schatzmann, Georgila, et al. - 2005