Although several real multimodal systems have been built, their development still remains a difficult task. In this paper we address this problem of development of multimodal interfaces by describing a component-based approach, called ICARE, for rapidly developing multimodal interfaces. ICARE stands for Interaction-CARE (Complementarity Assignment Redundancy Equivalence). Our component-based approach relies on two types of software components. Firstly ICARE elementary components include Device components and Interaction Language components that enable us to develop pure modalities. The second type of components, called Composition components, define combined usages of modalities. Reusing and assembling ICARE components enable rapid development of multimodal interfaces. We have developed several multimodal systems using ICARE and we illustrate the discussion using one of them: the FACET simulator of the Rafale French military plane cockpit.

Cameras are a useful source of input for many interactive applications, but computer vision programming is difficult and requires specialized knowledge that is out of reach for many HCI practitioners. In an effort to learn what makes a useful computer vision design tool, we created Eyepatch, a tool for designing camera-based interactions, and evaluated the Eyepatch prototype through deployment to students in an HCI course. This paper describes the lessons we learned about making computer vision more accessible, while retaining enough power and flexibility to be useful in a wide variety of interaction scenarios. ACM Classification: H.1.2 [Information Systems]:

Children with autism often exhibit self-stimulatory (or “stimming”) behaviors. We present an on-body sensing system for continuous recognition of stimming activity. By creating a system to recognize and monitor stimming behaviors, we hope to provide autism researchers with detailed, quantitative data. In this paper, we compare isolated and continuous recognition rates of emulated autistic stimming behaviors using hidden Markov models (HMMs). We achieved an overall system accuracy 68.57 % in continuous recognition tests. However, the occurrence of stimming events can be detected with 100 % accuracy by allowing minor frame–level insertion errors. 1

We present a classification of gesture-based computer interactions motivated by a literature review of over 40 years of gesture based interactions. This work presents a unique perspective on gesturebased interactions, categorized in terms of four key elements: gesture styles, the application domains they are applied to, input technologies and output technologies used for implementation. The classification provides a means of addressing gestures as an interaction mode across the different application domains so that researchers and designers can draw on the vast amount of research that has been addressed within the literature from an interaction perspective.

Identifying and recording subject movements is a critical, but time-consuming step in animal behavior research. The task is especially onerous in studies involving social insects because of the number of animals that must be observed simultaneously. To address this, we present a system that can automatically analyze animal movements, and label them, by creating a behavioral model from examples provided by a human expert. Further, in conjunction with identifying movements, our system also recognizes the behaviors made up of these movements. Thus, with only a small training set of hand labeled data, the system automatically completes the entire behavioral modeling and labeling process. For our experiments, activity in an observation hive is recorded on video, that video is converted into location information for each animal by a vision-based tracker, and then numerical features such as velocity and heading change are extracted. The features are used in turn to label the sequence of movements for each observed animal, according to the model. Our approach uses a combination of k-nearest neighbor (KNN) classification and hidden Markov model (HMM) techniques. The system was evaluated on several hundred honey bee trajectories extracted from a 15 minute video of activity in an observation hive. Additionally, simulated data and models were used to test the validity of the behavioral recognition techniques. 1.

Inspired by the Defense Advanced Research Projects Agency’s (DARPA) recent successes in speech recognition, we introduce a new task for sign language recognition research: a mobile one-way American Sign Language translator. We argue that such a device should be feasible in the next few years, may provide immediate practical benefits for the Deaf community, and leads to a sustainable program of research comparable to early speech recognition efforts. We ground our efforts in a particular scenario, that of a Deaf individual seeking an apartment and discuss the system requirements and our interface for this scenario. Finally, we describe initial recognition results of 94 % accuracy on a 141 sign vocabulary signed in phrases of fours signs using a one-handed glove-based system and hidden Markov models (HMMs). 1.

Abstract. Weight training, in addition to aerobic exercises, is an important component of a balanced exercise program. However, mechanisms for tracking free weight exercises have not yet been explored. In this paper, we study methods that automatically recognize what type of exercise you are doing and how many repetitions you have done so far. We incorporated a three-axis accelerometer into a workout glove to track hand movements and put another accelerometer on a user’s waist to track body posture. To recognize types of

Abstract. The Gesture and Activity Recognition Toolit (GART) is a user interface toolkit designed to enable the development of gesturebased applications. GART provides an abstraction to machine learning algorithms suitable for modeling and recognizing different types of gestures. The toolkit also provides support for the data collection and the training process. In this paper, we present GART and its machine learning abstractions. Furthermore, we detail the components of the toolkit and present two example gesture recognition applications.

Abstract. The Augmented Knight’s Castle is an augmented toy environment that enriches the children’s pretend play by using background music, sound effects, verbal commentary of toys, and different forms of tactile and visual feedback in reaction to the children’s play. Moreover, interactive learning experiences can be integrated into the play (e.g. to teach songs and poems or to provide the child with facts about the Middle Ages). We describe the different possibilities that are realized in our augmented playset, based on various mobile and pervasive computing technologies. Radio frequency identification (RFID) technology is used to automatically and unobtrusively identify toys in the playset. Mobile phones and “smart toys ” equipped with sensors and RFID readers are introduced into the playset to enhance the play and to provoke further interaction. 1

We address the feature selection problem for hidden Markov models (HMMs) in sequence classification. Temporal correlation in sequences often causes difficulty in applying feature selection techniques. Inspired by segmental k-means segmentation (SKS) [1], we propose Segmentally Boosted HMMs (SBHMMs), where the stateoptimized features are constructed in a segmental and discriminative manner. The contributions are twofold. First, we introduce a novel feature selection algorithm, where the temporal dynamics are decoupled from the static learning procedure by assuming that the sequential data are piecewise independent and identically distributed. Second, we show that the SBHMM consistently improves traditional HMM recognition in various domains. The reduction of error compared to traditional HMMs ranges from 17 % to 70 % in American Sign Language recognition, human gait identification, lip reading, and speech recognition.