We discuss current approaches to the development of natural language dialogue systems, and claim that they do not sufficiently consider the unique qualities of man-machine interaction as distinct from general human discourse. We conclude that empirical studies of this unique communication situation is required for the de-velopment of user-friendly interactive systems. One way of achieving this is through the use of so-called Wizard of Oz studies. We describe our work in this ar-ea. The focus is on the practical execution of the studies and the methodological conclusions that we have drawn on the basis of our experience. While the focus is on natural language interfaces, the methods used and the conclusions drawn from the results obtained are of relevance also to other kinds of intelligent interfaces. 1 THE NEED FOR WIZARD OF OZ STUDIES Dialogue has been an active research area for quite some time in natural language processing. It is fair to say that researchers studying dialogue and discourse have developed their theories through detailed analysis of empirical data from many di-

This paper describes a method for the development of dialogue managers for natural language interfaces. A dialogue manager is presented designed on the basis of both a theoretical investigation of models for dialogue management and an analysis of empirical material. It is argued that for natural language interfaces many of the human interaction phenomena accounted for in, for instance, plan-based models of dialogue do not occur. Instead, for many applications, dialogue in natural language interfaces can be managed from information on the functional role of an utterance as conveyed in the linguistic structure. This is modelled in a dialogue grammar which controls the interaction. Focus structure is handled using dialogue objects recorded in a dialogue tree which can be accessed through a scoreboard by the various modules for interpretation, generation and background system access. A sublanguage approach is proposed. For each new application the Dialogue Manager is customized to meet the...

Abstract not found

Our everyday use of the term situation suggests that it is unproblematic to decide what constitutes a situation in which language happens. At the same time it seems to be important to account for what a situation is like since situational factors have been found to influence systematically the properties of the language which occurs in that situation. We are investigating a corpus of (simulated) human-computer interaction in which the simulated system's utterances are produced independent of what the speaker says according to a fixed schema with recurring phases. This way, what one communication partner utters is both intra- and interpersonally comparable. The corpus thus provides a unique opportunity to study linguistic behaviour in a completely fixed situation in which even the contributions of one of the speakers can be controlled. Although situational variables are kept constant in this corpus, the speakers' linguistic behaviour varies both intra- and interpersonally. Given the hyp...

We address two problems in the field of automatic optimization of dialogue strategies: learning effective dialogue strategies when no initial data or system exists, and evaluating the result with real users. We use Reinforcement Learning (RL) to learn multimodal dialogue strategies by interaction with a simulated environment which is “bootstrapped ” from small amounts of Wizard-of-Oz (WOZ) data. This use of WOZ data allows development of optimal strategies for domains where no working prototype is available. We compare the RL-based strategy against a supervised strategy which mimics the wizards ’ policies. This comparison allows us to measure relative improvement over the training data. Our results show that RL significantly outperforms Supervised Learning when interacting in simulation as well as for interactions with real users. The RL-based policy gains on average 50-times more reward when tested in simulation, and almost 18-times more reward when interacting with real users. Users also subjectively rate the RL-based policy on average 10 % higher. 1

This paper describes MDWOZ, a development environment for spoken dialog systems based on the Wizard of Oz technique, whose main goal is to facilitate data collection (speech signal and dialog related information) and interaction model building. Both these tasks can be quite difficult, and such an environment can facilitate them very much. Due to the modular way in which MDWOZ was implemented, it is possible to reuse parts of it in the final dialog system. The environment provides language-transparent facilities and accessible methods such that even non-computing specialists can participate in spoken dialog systems development. The main features of the environment are presented, together with some test experiments. 1.

Models for predicting judgments about the quality of Spoken Dialog Systems have been used as overall evaluation metric or as optimization functions in adaptive systems. We describe a new approach to such models, using Hidden Markov Models (HMMs). The user’s opinion is regarded as a continuous process evolving over time. We present the data collection method and results achieved with the HMM model. 1

This paper presents a coding scheme which has been used for the analysis of NLI-dialogues collected by means of Wizard-of-Oz techniques. The scheme covers both dialogue structure and focus structure. Dialogue structure is coded in terms of segments consisting of moves belonging to general illocutionary types, such as initiative and response, and being further speci ed as to their topical domains. Focus structure is coded in terms of a number of focal parameters, which may di er from one type of dialogue to another. Relations between values of focal parameters on neighboring discourse segments are determined by a simple model. The coding scheme is exible and has a high degree of inter rater reliability. It enables comparisons between di erent types of dialogues and testing of assumed models for dialogue management. A dialogue manager has been implemented that can be customized on the basis of the analysis of a representative set of dialogues from a given application using the coding scheme.

This paper gives a characterisation of dialogue properties which are argued to be related to the usability of dialogue systems: effectiveness, effciency, transparency and a number of properties that have to do with coherence. Such properties can be used as guidelines in dialogue design, but also as aspects to test for in verification and validation.

From a system developer’s perspective, designing a spoken dialogue system can be a time-consuming and difficult process. A developer may spend a lot of time anticipating how a potential user might interact with the system and then deciding on the most appropriate system response. These decisions are encoded in a dialogue strategy, essentially a mapping between anticipated user inputs and appropriate system outputs. To reduce the time and effort associated with developing a dialogue strategy, recent work has concentrated on modelling the development of a dialogue strategy as a sequential decision problem. Using this model, reinforcement learning algorithms have been employed to generate dialogue strategies automatically. These algorithms learn strategies by interacting with simulated users. Some progress has been made with this method but a number of important challenges remain. For instance, relatively little success has been achieved with the large state representations that are typical of real-life systems. Another crucial issue is the time and effort associated with the creation of simulated users.