In this paper, we describe an implementation of a plan-based multimodal dialog system using the extensible markup language (XML). The dialog manager receives semantic objects representing the user’s utterance at the end of each user’s turn. We define a semantic markup language (SML), based on XML, to describe these semantic objects. Following the principles of XML Schema, we define the schema of SML in another XML called semantic definition language (SDL). In addition to supporting many discourse and dialog features, SDL is also designed to represent the domain knowledge via the application schema and the hierarchy of the semantic objects. We show that, with a thoughtful design in SDL, SML can be expressive enough that the behavior of a dialog planner can be fully specified in the extensible stylesheet language (XSL), a standardized language with a logical programming model that is most popular for implementing intelligent systems.

SALT is an industrial standard that enables speech input/output for Web applications. Although the core design is to make simple tasks easy, SALT gives the designers ample fine-grained controls to create advanced user interface. The paper exploits a speech input mode in which SALT would dynamically report partial semantic parses while audio capturing is still ongoing. The semantic parses can be evaluated and the outcome reported immediately back to the user. The potential impact for the dialog systems is that tasks conventionally performed in a system turn can now be carried out in the midst of a user turn, thereby presenting a significant departure from the conventional turn-taking. To assess the efficacy of such highly interactive interface, more user studies are undoubtedly needed. This paper demonstrates how SALT can be employed to facilitate such studies.