Abstract

We consider the problem of multimodal message generation. The main ideas of this paper are to unify the treatment of speech, text and graphics using a single grammar, and to adapt an evaluation function used to estimate the subjective quality of proposed messages by observing the time it takes the user to respond to them. 1 Introduction The work described in this paper is part of a larger ongoing effort by the Duke Voice and Natural Language Lab to develop speech-graphics interactive dialogue systems [2, 7, 1] built around a theorem proving mechanism. The system attempts to prove that some global goal is achievable, and if it is successful, no significant dialogue takes place. However, if the system finds that the global goal cannot be proved, it searches for "missing axioms" which the user might be able to help the system achieve and which would make the global goal provable. It then initiates speech-graphics interactions with the user to try to prove those missing axioms. This paper...

Citations