This work shows how a dialogue model can be represented as a Partially Observable Markov Decision Process (POMDP) with observations composed of a discrete and continuous component. The continuous component enables the model to directly incorporate a confidence score for automated planning. Using a testbed simulated dialogue management problem, we show how recent optimization techniques are able to find a policy for this continuous POMDP which outperforms a traditional MDP approach. Further, we present a method for automatically improving handcrafted dialogue managers by incorporating POMDP belief state monitoring, including confidence score information. Experiments on the testbed system show significant improvements for several example handcrafted dialogue managers across a range of operating conditions. 1

The need for N a ur a La ngua' Interfa3; to da ta ba ses (NLIs) ha s become increa3] glya cute a more a d more peoplea ccess informa ion through their web browsers, PDAs, a d cell phones.Yet NLIsa re only usa ble if theyma p na tura la gu a. questions to SQL queries correctly As Schneiderma a d Norma ha vea rgued, people a. unwilling to tr a e relia lea nd predicta ble user interfaMM for intelligent but unrelia ble ones In thispa per, we introducea theoretica l fra mework for relia ble NLIs, which is the founda tion for the fully implemented Precise NLI We prove tha t, fora broa cla3 of semantically tractable tura la gu a. questions, Precise is gua.' teed to ma p e a h question to the corresponding SQL query We report on experiments testing Precise on severa l hundred questions dra wn from user studies over three benchma rkda ta ba ses We findtha t over 80% of the questionsa' sem a tica;M traE a ble questions, which Precise a swers correctly lly recognizes the 20% of questions tha t it c a not h a dle, a d requestsa pa ra phra se Fina lly, we show tha t Precise compa res fa vora ly with Mooney'slea: ing NLIa nd with Microsoft 's English Query product Categories and Subject Descriptors H 2 3 [Database Management]: Query L a gua.E3 SQL; H 5 2 [Information Interfaces and Presentation]: User Interfa:M , N a ur a La ngua' General Terms gua:------ Algorithms, Relia ility Keywords tura la gu a. interfa'H da ta ba se, relia ility # We th a k Keith Golden, D a Weld, Bonnie Weber a d Tessa La for comments on previous dra' s Thisresea] h wa s supported in pa' by ONRgra t N00014-02-1-0324 Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for prof...

We propose a method for learning dialogue management policies from a fixed dataset. The method is designed for use with "Information State Update " (ISU)-based dialogue systems, which represent the state of a dialogue as a large set of features, resulting in a very large state space and a very large policy space. To address the problem that any fixed dataset will only provide information about small portions of these state and policy spaces, we propose a hybrid model which combines reinforcement learning (RL) with supervised learning.

... this paper, we present a new type of recommendation system that carries out a personalized dialogue with the user. This system -- the Adaptive Place Advisor -- treats item selection as an interactive, conversational process, with the program inquiring about item attributes and the user responding. The system incorporates a user model that contains item, attribute, and value preferences, which it updates during each conversation and maintains across sessions. The Place Advisor uses both the conversational context and the user model to retrieve candidate items from a case base. The system then continues to ask questions, using personalized heuristics to select which attribute to ask about next, presenting complete items to the user only when a few remain. We report experimental results demonstrating the effectiveness of user modeling in reducing the time and number of interactions required to find a satisfactory item

and reinforcement learning

This work shows how a dialogue model can be represented as a factored Partially Observable Markov Decision Process (POMDP). The factored representation has several benefits, such as enabling more nuanced reward functions to be specified. Although our dialogue model is significantly larger than past work using POMDPs, experiments on a small testbed problem demonstrate that recent optimisation techniques scale well and produce policies which outperform a traditional fully-observable Markov Decision Process. This work then shows how a dialogue manager produced with a POMDP optimisation technique may be directly compared to a handcrafted dialogue manager. Experiments on the testbed problem show that automatically generated dialogue managers outperform several handcrafted dialogue managers, and that automatically generated dialogue managers for the testbed problem successfully adapt to changes in speech recognition accuracy. 1

Statistical methods have long been the dominant approach in speech recognition and probabilistic modelling in ASR is now a mature technology. The use of statistical methods in other areas of spoken dialogue is however more recent and rather less mature. This paper reviews spoken dialogue systems from a statistical modelling perspective. The complete system is first presented as a partially observable Markov decision process. The various sub-components are then exposed by introducing appropriate intermediate variables. Samples of existing work are reviewed within this framework, including dialogue control and optimisation, semantic interpretation, goal detection, natural language generation and synthesis.

Partially observable Markov decision processes (POMDPs) provide a principled mathematical framework for modelling the uncertainty inherent in spoken dialog systems. However, conventional POMDPs scale poorly with the size of state and observation space. This paper describes a variation of the classic POMDP called the Hidden Information State (HIS) model in which belief distributions are represented efficiently by grouping states together into partitions and policy optimisation is made tractable by using a master to summary space mapping. An implementation of the HIS model is described for a Tourist Information application and aspects of its training and operation are illustrated. Index Terms — statistical dialog modelling; partially observable Markov decision processes (POMDPs) 1.

We propose the “advanced ” n-grams as a new technique for simulating user behaviour in spoken dialogue systems, and we compare it with two methods used in our prior work, i.e. linear feature combination and “normal ” n-grams. All methods operate on the intention level and can incorporate speech recognition and understanding errors. In the linear feature combination model user actions (lists of 〈 speech act, task 〉 pairs) are selected, based on features of the current dialogue state which encodes the whole history of the dialogue. The user simulation based on “normal ” n-grams treats a dialogue as a sequence of lists of 〈 speech act, task 〉 pairs. Here the length of the history considered is restricted by the order of the n-gram. The “advanced ” n-grams are a variation of the normal ngrams, where user actions are conditioned not only on speech acts and tasks but also on the current status of the tasks, i.e. whether

We explore the use of restricted dialogue contexts in reinforcement learning (RL) of effective dialogue strategies for information seeking spoken dialogue systems (e.g. COMMUNICATOR (Walker et al., 2001)). The contexts we use are richer than previous research in this area, e.g. (Levin and Pieraccini, 1997; Scheffler and Young, 2001; Singh et al., 2002; Pietquin, 2004), which use only slot-based information, but are much less complex than the full dialogue “Information States ” explored in (Henderson et al., 2005), for which tractabe learning is an issue. We explore how incrementally adding richer features allows learning of more effective dialogue strategies. We use 2 user simulations learned from COMMUNICATOR data (Walker et al., 2001; Georgila et al., 2005b) to explore the effects of different features on learned dialogue strategies. Our results show that adding the dialogue moves of the last system and user turns increases the average reward of the automatically learned strategies by 65.9 % over the original (hand-coded) COMMUNI-CATOR systems, and by 7.8 % over a baseline RL policy that uses only slot-status features. We show that the learned strategies exhibit an emergent “focus switching” strategy and effective use of the ‘give help ’ action. 1