DyPERS, 'Dynamic Personal Enhanced Reality System', is a wearable system which uses augmented reality and computer vision to autonomously retrieve 'media memories' based on associations with real objects the user encounters. These are evoked as audio and video clips taken by the user and overlayed on top of real objects the user looks at. The user's visual and auditory scene is stored in real-time by the system (upon request) and is then associated (by user input) with a snap shot of a visual object. The object acts as a key which is detected by a real-time vision system when it is in view, triggering DyPERS to playback the appropriate audio-visual sequence. The vision system is a probabilistic algorithm which is capable of discriminating between hundreds of everyday objects under varying viewing conditions (lighting, pose, etc.). The record-and-associate paradigm of the system has many potential applications. Results on the use of the system in a museum tour scenario are described.

Recent advances in various display and virtual technologies coupled with an explosion in available computing power have given rise to a numberofnovel human-computer interaction (HCI) modalities -- speech, vision-based gesture recognition, eye tracking, EEG, etc. However, despite the abundance of novel interaction devices, the naturalness and efficiency of HCI has remained low. This is due in particular to the lack of robust sensory data interpretation techniques. To deal with the task of interpreting single and multiple interaction modalities this dissertation establishes a novel probabilistic approach based on dynamic Bayesian networks (DBNs). As a generalization of the successful hidden Markov models, DBNs are a natural basis for the general temporal action interpretation task. The problem of interpretation of single or multiple interacting modalities can then be viewed as a Bayesian inference task. In this work three complex DBN models are introduced: mixtures of DBNs, mixed-state DBNs, and coupled HMMs. In-depth study of these models yields efficient approximate inference and parameter learning techniques applicable to a wide variety of problems. Experimental validation of the proposed approaches in the domains of gesture and speech recognition con rms the model's applicability to both unimodal and multimodal interpretation tasks.

Introduction Over the past decade, there has been a ground swell of activityintwo #elds of user interface research: augmented reality and wearable computing. Augmentedreality #1# refers to the creation of virtual environments that supplement, rather than replace, the real world with additional information. This is accomplished through the use of #see-through" displays that enrich the user's view of the world byoverlaying visual, auditory, and even haptic, material on what she experiences. Visual augmented reality systems typically, but not exclusively, employ head-tracked, head-worn displays. These either use half-silvered mirror beam splitters to re#ect small computer displays, optically combining them with a view of the real world, or use opaque displays fed by electronics that merge imagery captured by head-worn cameras with synthesized graphics. Wearable computing moves computers o# the desktop and onto the user's body, made possible through the

This paper describes an algorithm to detect and track a hand in each image taken by a wearable camera. We primarily use color information, however, instead of prede ned skin-color models, we dynamically construct hand- and background-color models by using a Gaussian mixture model to approximate the color histogram. Not only to obtain the estimated mean of hand color necessary for the restricted EM algorithm that estimates the GMM but to classify hand pixels based on the Bayes decision theory, we use a spatial probability distribution of hand pixels. Because the static distribution is inadequate for the hand-tracking stage, we translate the distribution with the hand motion based onthe mean shift algorithm. Using the proposed method, we implemented the Hand Mouse, that uses the wearer's hand as a pointing device, on our Wearable Vision System.

Gesture interaction is a versatile, intuitive way of interacting with computers, especially suited for virtual environments and applications where many degrees of freedom are important. Human gestures have many meanings and uses, and are often different from person to person, while gesture input devices often introduce artifacts; this makes recognizing and interpreting gestures nontrivial. Gestures can be divided into postures, where the configuration of the hand is static, and true gestures, where the flexion of the fingers and the hand position/direction are changing dynamically. A recognition model based on a hybrid articial neural network combining a radial basis function and a Bayesian classifier network is developed and tested with various forms of pre-processing or input representation. The results suggest that dynamic gesture recognition is feasible even for complex gestures, but that context information may be needed for reliable recognition.

1 Motivation for contextual aware computing: For most computer systems, even virtual reality systems, sensing techniques are a means of getting input directly from the user. However, wearable sensors and computers offer a unique opportunity to re-direct sensing technology towards recovering more general user context. Wearable computers have the potential to "see" as the user sees, "hear" as the user hears, and experience the life of the user in a "first-person" sense. This increase in contextual and user information may lead to more intelligent and fluid interfaces that use the physical world as part of the interface. Wearable computers are excellent platforms for contextually aware applications, but these applications are also necessary to use wearables to their fullest. Wearables are more than just highly portable computers, they perform useful work even while the wearer isn't directly interacting with the system. In such environments the user needs to concentrate on his environment, not on the computer interface, so the wearable needs to use information from the wearer's context to be the least distracting. For example, imagine an interface which is aware of the user's location: while being in the subway, the system might alert him with a

This paper describes a wearable input interface the functions of which is similar to the usual mouse. The Hand-mouse is realized by classifying each pixel as a hand pixel or a background pixel based on approximation of a color histogram with the Gaussian mixture model and by tting the simple model of hand shapes into the classi ed hand pixels.