Abstract not found

Creating high-quality multimedia presentations requires much skill, time, and effort. This is particularly true when temporal media, such as speech and animation, are involved. We describe the design and implementation of a knowledge-based system that generates customized temporal multimedia presentations. We provide an overview of the system’s architecture, and explain how speech, written text, and graphics are generated and coordinated. Our emphasis is on how temporal media are coordinated by the system through a multi-stage negotiation process. In negotiation, media-specific generation components interact with a novel coordination component that solves temporal constraints provided by the generators. We illustrate our work with a set of examples generated by the system in a testbed application intended to update hospital caregivers on the status of patients who have undergone a cardiac bypass operation.

ion Signal Level Semantic Level Figure 13: Multimodal Design Space (adapted from [224]) system in the design space is the pivotal center of its features. According to the characterization of an interaction along the two dimensions, fusion, and use of modalities, four basic types of multimodal interactions can be distinguished: alternative, synergistic, exclusive, and concurrent multimodal interaction, as shown in Figure 13. Obviously, synergistic systems subsume the other three classes of multimodal systems. Therefore, architectural models of multimodal integration (as presented in the next subsection and in Section 9) are sufficient if they are able to model synergistic cooperation of modalities. 6.2.2 Fusion of Multimodal Input Fusion of multimodal input events can occur on different levels, ranging from signal-level to semantic-level. Signal-level fusion (or lexical fusion [224]) performs the combination of multimodal input at the level of the input signal. Signal-level fusion has...

goes to our extended research team, especially David McNeill, a friend and colleague, upon whose psycholinguistic research this work is based. The Catchment Feature Model addresses two questions in the field of multimodal interaction: how do we bridge video and audio processing with the realities of human multimodal communication, and how information from the different modes may be fused. We argue from a detailed literature review that gestural research has clustered around manipulative and semaphoric use of the hands, motivate the Catchment Feature Model psycholinguistic research, present the Model. In contrast to ‘whole gesture ’ recognition, the Catchment Feature Model applies a feature decomposition ap-proach that facilitates cross-modal fusion at the level of discourse planning and conceptualization. We present our experimental framework for catchment feature-based research, and cite three con-crete examples of Catchment Features, and propose new directions of multimodal research based on the model. 1

Facial modeling and animation are increasingly receiving attention in the graphics and artificial intelligence (AI) research communities, both of which share the common goal of synthesizing believable, simulated agents. While computer graphics researchers have been primarily concerned with physical and anatomical aspects of facial movements, AI researchers and cognitive scientists have focused on understanding and modeling the motivation behind those movements and expressions. The combination of these two avenues of research may eventually lead to agents that can interact autonomously, with humans or with each other, bearing faces that believably model the underlying meaning of the interactions. While such synthetic speaking faces are undoubtedly useful for cognitive research, their practical applications are also vast in number, encompassing such diverse fields as medicine, education, telecommunications and the entertainment industry. Facial expressions have fascinated mankind for ce...