In this paper, we present research toward ontology-based understanding of discourse in meetings and describe an ontology of multimodal discourse designed for this purpose. We investigate its application in an integrated but modular architecture which uses semantically annotated knowledge of communicative meeting activity as well as discourse subject matter. We highlight how this approach assists in improving system performance over time and supports understanding in a changing and persistent environment. We also describe current and future plans for ontology-driven robust naturallanguage understanding in the presence of the highly ambiguous and errorful input typical of the meeting domain.

A problem faced by groups that are not co-located but need to collaborate on a common task is the reduced access to the rich multimodal communicative context that they would have access to if they were collaborating face-to-face. Collaboration support tools aim to reduce the adverse effects of this restricted access to the fluid intermixing of speech, gesturing, writing and sketching by providing mechanisms to enhance the awareness of distributed participants of each others ’ actions. In this work we explore novel ways to leverage the capabilities of multimodal context-aware systems to bridge colocated and distributed collaboration contexts. We describe a system that allows participants at remote sites to collaborate in building a project schedule via sketching on multiple distributed whiteboards, and show how participants can be made aware of naturally occurring pointing gestures that reference diagram constituents as they are performed by remote participants. The system explores the multimodal fusion of pen, speech and 3D gestures, coupled to the dynamic construction of a semantic representation of the interaction, anchored on the sketched diagram, to provide feedback that overcomes some of the intrinsic ambiguities of pointing gestures.

In groupware, users must communicate about their intentions and maintain common knowledge via communication channels that are explicitly designed into the system. Depending upon the task, generic communication tools like chat or a shared whiteboard may not be sufficient to support effective coordination. We have previously reported on a methodology that helps the designer develop task specific communication tools, called coordinating representations, for groupware systems. Coordinating representations lend structure and persistence to coordinating information. We have shown that coordinating representations are readily adopted by a user population, reduce coordination errors, and improve performance in a domain task. As we show in this article, coordinating representations present a unique opportunity to acquire user information in collaborative, user-adapted systems. Because coordinating representations support the exchange of coordinating information, they offer a window onto task and coordination-specific knowledge that is shared by users. Because they add structure to communication, the information that passes through them can be easily exploited by adaptive technology. This approach provides a simple technique for acquiring user knowledge in collaborative, user-adapted systems. We document our application of this approach to an existing groupware system. Several empirical results are

It is tedious to handwrite long passages of text by hand. To make this process more efficient, we propose predictive handwriting that provides input predictions when the user writes by hand. A predictive handwriting system presents possible next words as a list and allows the user to select one to skip manual writing. Since it is not clear if people are willing to use prediction, we first run a user study to compare handwriting and selecting from the list. The result shows that, in Japanese, people prefer to select, especially when the expected performance gain from using selection is large. Based on these observations, we designed a multimodal input system, called speech-pen, that assists digital writing during lectures or presentations with background speech and handwriting recognition. The system recognizes speech and handwriting in the background and provides the instructor with predictions for further writing. The speech-pen system also allows the sharing of context information for predictions among the instructor and the audience; the result of the instructor’s speech recognition is sent to the audience to support their own note-taking. Our preliminary study shows the effectiveness of this system and the implications for further improvements. Author Keywords Predictive handwriting, speech recognition, handwriting

A basic capability driving the development of systems is the ability to execute multiple versions of a system against the same set of data. That is essential for instance to verify that coding problems were corrected, to provide guarantees that changes to the code did not introduce errors (regression testing) and for comparison of performance among different algorithmic solutions. Multimodal systems process a combination of inputs and their results are susceptible to timing issues, which determine for instance whether a pair of inputs from different modalities combine or not. When a multimodal system is executed against corpus data, it becomes necessary to synchronize processing of the various input streams and to handle time-related information so as to emulate real-time execution. This is particularly complex when multiple components of the various input stream processors take longer than real-time. In this paper we describe a multimodal play back mechanism that addresses these problems. We report on two data collection and play back software and hardware environments — one used for system development and evaluation and the other used for supporting manual annotation of multimodal data.

Two important themes in current work on interfaces are multimodal interaction and the use of dialogue. Human multimodal dialogues are symmetric, i.e., both participants communicate multimodally. We describe a proof of concept system that supports symmetric multimodal communication for speech and sketching in the domain of simple mechanical device design. We discuss three major aspects of the communication: multimodal input processing, multimodal output generation, and creating a dynamic dialogue. While previous systems have had some of these capabilities individually, their combination appears to be unique. We provide examples from our system that illustrate a variety of user inputs and system outputs. Author Keywords multimodal, dynamic dialogue, sketch recognition, sketch generation, speech ACM Classification Keywords

Recent years have witnessed an increasing shift in interest from single user multimedia/multimodal interfaces towards support for interaction among groups of people working closely together, e.g. during meetings or problem solving sessions. However, the introduction of technology to support collaborative practices has not been devoid of problems. It is not uncommon that technology meant to support collaboration may introduce disruptions and reduce group effectiveness. Human-centered multimedia and multimodal approaches hold a promise of providing substantially enhanced user experiences by focusing attention on human perceptual and motor capabilities, and on actual user practices. In this paper we examine the problem of providing effective support for collaboration, focusing on the role of human-centered approaches that take advantage of multimodality and multimedia. We show illustrative examples that demonstrate human-centered multimodal and multimedia solutions that provide mechanisms for dealing with the intrinsic complexity of human-human interaction support.

In this paper, we present research toward ontology-based understanding of discourse in meetings and describe an ontology of multimodal discourse designed for this purpose. We investigate its application in an integrated but modular architecture which uses semantically annotated knowledge of communicative meeting activity as well as discourse subject matter. We highlight how this approach assists in improving system performance over time and supports understanding in a changing and persistent environment. We also describe current and future plans for ontology-driven robust naturallanguage understanding in the presence of the highly ambiguous and errorful input typical of the meeting domain.

Brief Statement of the Problem: Sketching is used in early stage design, but it doesn’t capture or provide the user all the avenues of communication that talking to a design partner does. By creating an interface that combines speech and sketching, we hope to engage the user more fully and capture and understand more of the sketch in a manner similar to a human design partner. We envision a multimodal interactive dialogue. As the user talks and sketches, the computer attempts to understand. It can ask questions to resolve uncertainties, inquire about vocabulary, or ask about similar components in the sketch. These questions could be asked using a combination of verbal and sketched output. We have conducted user studies to gather data about how two people naturally interact during design discussions, and will use the results to help guide the design of our system.

Sketching is commonly used in the early stages of design, however, some information is difficult to express using sketching alone. When designers talk while sketching, a considerable amount of information is also conveyed using