We describe an implemented system which automatically generates and animates conversations between multiple human-like agents with appropriate and synchronized speech, intonation, facial expressions, and hand gestures. Conversations are created by a dialogue planner that produces the text as well as the intonation of the utterances. The speaker/listener relationship, the text, and the intonation in turn drive facial expressions, lip motions, eye gaze, head motion, and arm gesture generators. Coordinated arm, wrist, and hand motions are invoked to create semantically meaningful gestures. Throughout, we will use examples from an actual synthesized, fully animated conversation. 1

This paper reports results from a program that produces high quality animation of facial expressions and head movements as automatically as possible in conjunction with meaning-based speech synthesis, including spoken intonation. The goal of the research is as much to test and define our theories of the formal semantics for such gestures, as to produce convincing animation. Towards this end we have produced a high level programming language for 3D animation of facial expressions. We have been concerned primarily with expressions conveying information correlated with the intonation of the voice: this includes the differences of timing, pitch, and emphasis that are related to such semantic distinctions of discourse as “focus”, “topic ” and “comment”, “theme ” and “rheme”, or “given ” and “new ” information. We are also interested in the relation of affect or emotion to facial expression. Until now, systems have not embodied such rule-governed translation from spoken utterance meaning to facial expressions. Our system embodies rules that describe and coordinate these relations: intonation/information, intonation/affect and facial expressions/affect. A meaning representation includes discourse information: what is contrastive/background information in the given context, and what is the “topic ” or “theme ” of the discourse. The system maps the meaning representation into how accents and their placement are chosen, how they are conveyed over facial expression and how speech and facial expressions are coordinated. This determines a sequence of functional groups: lip shapes, conversational signals, punctuators, regulators or manipulators. Our algorithms then impose synchrony, create coarticulation effects, and determine affectual signals, eye and head movements. The lowest level representation is the Facial Action Coding System (FACS), which makes the generation system portable to other facial models.

User-interruption in human-computer interaction (HCI) is an increasingly important problem. Many of the useful advances in intelligent and multitasking computer systems have the significant side effect of greatly increasing user-interruption. This previously innocuous HCI problem has become critical to the successful function of many kinds of modern computer systems. Unfortunately, no HCI design guidelines exist for solving this problem. In fact, theoretical tools do not yet exist for investigating the HCI problem of user-interruption in a comprehensive and generalizable way. This report asserts that a single unifying definition of user-interruption and the accompanying practical taxonomy would be useful theoretical tools for driving effective investigation of this crucial HCI problem. These theoretical tools are constructed here. A comprehensive analysis is conducted through the existing literature. Theoretical constructs from several relevant but diverse fields are identified and discussed. A unifying definition of user-interruption is synthesized. This new definition is supported with an array of postulates, assertions, and a taxonomy of human interruption to facilitate its practical application.

). I.."how la operate a'8ice and USCl LUI"f"L.... J cd ments, & Computers!l'6, 249-255. pohon, P. G., & L,wi " C. H. (1990). Tb,my-b""d d"ign fo ',,,,ily ',"m

We present a model of dialogue for embodied virtual agents that can communicate with multiple (human and virtual) agents in a multi-modal setting, including face-to-face spoken and nonverbal, as well as radio interaction, spanning multiple conversations in support of an extended complex task.

This paper reports on the first of a series of analyses aimed at comparing same room and video-mediated conversations for multiparty meetings. This study compared patterns of spontaneous speech for same room versus two videomediated conversations. One video system used a single camera, monitor and speaker, and a picture-in-a-picture device to display multiple people on one screen. The other system used multiple cameras, monitors, and speakers in order to support directional gaze cues and selective listening. Differences were found between same room and video-mediated conversations in terms of floor control and amount of simultaneous speech. While no differences were found between the video systems in terms of objective speech measures, other important differences are suggested and discussed.

Only humans communicate using language and carry on conversations with one another. And the skills of conversation have developed in humans in such a way as to exploit all of the unique affordances of the human body. We make complex representational gestures with our prehensile hands, gaze away and towards one another out of the corners of our centrally set eyes, and use the pitch and melody of our voices to emphasize and clarify what we are saying. Perhaps because conversation is so defining of humanness and human interaction, the metaphor of face-to-face conversation has been applied to human-computer interface design for quite some time. One of the early arguments for the utility of this metaphor gave a list of features of face-to-face conversation that could be applied fruitfully to human-computer interaction, including mixed initiative, nonverbal communication, sense of presence, rules for transfer of control (Nickerson 1976). However, although these feature

Abstract not found

In this paper, we discuss why, in designing multiparty mediated systems, we should focus first on providing non-verbal cues which are less redundantly coded in speech than those normally conveyed by video. We show how conveying one such cue, gaze direction, may solve two problems in multiparty mediated communication and collaboration: knowing who is talking to whom, and who is talking about what. As a candidate solution, we present the GAZE Groupware System, which combines support for gaze awareness in multiparty mediated communication and collaboration with small and linear bandwidth requirements. The system uses an advanced, deskmounted eyetracker to metaphorically convey gaze awareness in a 3D virtual meeting room and within shared documents. KEYWORDS: CSCW, multiparty videoconferencing, awareness, attention, gaze direction, eyetracking, VRML 2. INTRODUCTION With recent advances in network infrastructure and computing power, desktop video conferencing and groupware systems are ra...

At first glance it seems absurd that busy people doing important jobs should want their computers to interrupt them. Interruptions are disruptive and people need to concentrate to make good decisions. However, successful job performance also frequently depends on people's abilities to (a) constantly monitor their dynamically changing information environments, (b) collaborate and communicate with other people in the system, and (c) supervise background autonomous services. These critical abilities can require people to simultaneously query a large set of information sources, continuously monitor for important events, and respond to and communicate with other human operators. Automated monitoring