Data-driven techniques have been used for many computational linguistics tasks. Models derived from data are generally more robust than hand-crafted systems since they better reflect the distribution of the phenomena being modeled. With the availability of large corpora of spoken dialog, dialog management is now reaping the benefits of data-driven techniques. In this paper, we compare two approaches to modeling subtask structure in dialog: a chunk-based model of subdialog sequences, and a parse-based, or hierarchical, model. We evaluate these models using customer agent dialogs from a catalog service domain. 1

Computer Speech and Language xxx (2009) xxx–xxx COMPUTER SPEECH AND LANGUAGE www.elsevier.com/locate/csl

During the last decade, research in the field of Spoken Dialogue Systems (SDS) has experienced increasing growth. However, the design and optimization of SDS is not only about combining speech and language processing systems such as Automatic Speech Recognition (ASR), parsers, Natural Language Generation (NLG), and Text-to-Speech (TTS) synthesis systems. It also requires the development of dialogue strategies taking at least into account the performances of these subsystems (and others), the nature of the task (e.g. form filling, tutoring, robot control, or database search/browsing), and the user’s behaviour (e.g. cooperativeness, expertise). Due to the great variability of these factors, reuse of previous hand-crafted designs is also made very difficult. For these reasons, statistical machine learning (ML) methods applied to automatic SDS optimization have been a leading research area for the last few years. In this paper, we provide a short review of the field and of recent advances.

Intelligent planning algorithms such as the Partially Observable Markov Decision Process (POMDP) have succeeded in dialog management applications [10, 11, 12] because of their robustness to the inherent uncertainty of human interaction. Like all dialog planning systems, however, POMDPs require an accurate model of the user (the different states of the user, what the user might say, etc.). POMDPs are generally specified using a large probabilistic model with many parameters; these parameters are difficult to specify from domain knowledge, and gathering enough data to estimate the parameters accurately a priori is expensive. In this paper, we take a Bayesian approach to learning the user model simultaneously the dialog management problem. At the heart of our approach is an efficient incremental update algorithm that allows the dialog manager to replan just long enough to improve the current dialog policy given data from recent interactions. The update process has a relatively small computational cost, preventing long delays in the interaction. We are able to demonstrate a robust dialog manager that learns from interaction data, out-performing a hand-coded model in simulation and in a robotic wheelchair application.

Based on the framework of partially observable Markov decision processes (POMDPs), this paper describes a practical real-time spoken dialogue system in which the underlying belief state is represented by a dynamic Bayesian Network and the policy is parameterized using a set of action-dependent basis functions. Tractable real-time Bayesian belief updating is made possible using a novel form of Loopy Belief Propagation and policy optimisation is performed using an episodic Natural Actor Critic algorithm. Details of these algorithms are provided along with evaluations of their accuracy and efficiency. The proposed POMDP-based architecture was tested using both simulations and a user trial. Both indicated that the incorporation of Bayesian belief updating significantly increases robustness to noise compared to traditional dialogue state estimation approaches. Furthermore, policy learning worked effectively and the learned policy outperformed all others on simulations. In user trials the learned policy was also competitive, although its optimality was less conclusive. Overall, the Bayesian update of dialogue state framework was shown to be a feasible and effective approach to building real-world POMDP-based dialogue systems.

Abstract — Control in spoken dialog systems is challenging largely because automatic speech recognition is unreliable and hence the state of the conversation can never be known with certainty. Partially observable Markov decision processes (POMDPs) provide a principled mathematical framework for planning and control in this context; however, POMDPs face severe scalability challenges and past work has been limited to trivially small dialog tasks. This paper presents a novel POMDP optimization technique – composite summary point-based value iteration (CSPBVI) – which enables optimization to be performed on slot-filling POMDP-based dialog managers of a realistic size. Using dialog models trained on data from a tourist information domain, simulation results show that CSPBVI scales effectively, outperforms non-POMDP baselines, and is robust to estimation errors. Index Terms — Decision theory, dialogue management, partially observable Markov decision process, planning under uncertainty, spoken dialogue system. I.

This paper presents a new framework for accumulating beliefs in spoken dialogue systems. The technique is based on updating a Bayesian Network that represents the underlying state of a Partially Observable Markov Decision Process (POMDP). POMDP models provide a principled approach to handling uncertainty in dialogue but generally scale poorly with the size of the state and action space. The framework proposed, on the other hand, scales well and can be extended to handle complex dialogues. Learning is achieved with a factored summarising function that is applicable for many slot-filling type dialogues. The framework also provides a good structure from which to build hand-crafted policies. For very complex dialogues, this allows the POMDP’s principled approach to uncertainty to be incorporated without requiring computationally intensive learning algorithms. Simulations show that the proposed framework outperforms standard techniques whenever errors increase. Index Terms — Learning systems, Speech processing, Robustness 1.

The benefit of tracking a probability distribution over multiple dialogue states has been demonstrated in the literature. However, the dialogue state in past work has been limited to a small number of variables, and growing the number of variables in the dialogue state prevents the probability distribution from being updated in real-time. This paper shows how the number of variables composing the dialogue state can be increased while maintaining response times suitable for a spoken dialogue system. Rather than performing exact inference using the joint distribution over all variables, a particle filter is employed to compute an approximate update. Dialogue states (particles) are sampled, weighted by their agreement with the speech recognition results, and marginalized to produce a new distribution over each variable. Results on a spoken dialogue system for troubleshooting show that a relatively small number of particles are required to achieve performance close to an exact update, enabling the dialogue system to run in realtime. Index Terms — dialogue modelling, dialogue management, spoken dialogue systems, particle filter, Monte Carlo 1.

This paper reports on progress applying partially observable Markov decision processes (POMDPs) to a commercial dialog domain: troubleshooting. In the troubleshooting domain, a spoken dialog system helps a user to fix a product such as a failed DSL connection. Past work has argued that a POMDP is a principled approach to building spoken dialog systems in the simpler slot-filling domain; this paper explains how the POMDPs formulation can be extended to the more complex troubleshooting domain. Results from dialog simulation verify that a POMDP outperforms a handcrafted baseline. 1

Partially observable Markov decision processes (POMDPs) and conventional design practices offer two very different but complementary approaches to building spoken dialog systems. Whereas conventional manual design readily incorporates business rules, domain knowledge, and contextually appropriate system language, POMDPs employ optimization to produce more detailed dialog plans and better robustness to speech recognition errors. In this paper we propose a novel method for integrating these two approaches, capturing both of their strengths. The POMDP and conventional dialog manager run in parallel; the conventional dialog manager nominates a set of one or more actions, and the POMDP chooses the optimal action. Experiments using a real dialog system confirm that this unified architecture yields better performance than using a conventional dialog manager alone, and also demonstrate an improvement in optimization speed and reliability vs. a pure POMDP. Index Terms: dialogue modelling, dialogue management, spoken dialogue systems, partially observable Markov decision