this paper we present a particle filter-based visual tracker that fuses three cues in a novel way: color, motion, and sound (Fig. 1). More specifically, we will introduce color as the main visual cue and fuse it, depending on the scenario under consideration, with either sound localization cues or motion activity cues. The generic objective is to track a specified object or region of interest in the sequence of images captured by the camera. We employ weak object models so as not to be too restrictive about the types of objects the algorithm can track, and to achieve robustness to large variations in the object pose, illumination, motion, etc. In this generic context, contour cues are less appropriate than color cues to characterize the visual appearance of tracked entities. The use of edge-based cues indeed requires that the class of objects to be tracked is known a priori and that rather precise silhouette models can be learned beforehand. Note however that such conditions are met in a number of tracking applications where shape cues are routinely used [2], [3], [25], [30], [40], [44], [53]

A development of a method for tracking visual contours is described. Given an "un-trained" tracker, a training-motion of an object can be observed over some extended time and stored as an image sequence. The image sequence is used to learn parameters in a stochastic differential equation model. These are used, in turn, to build a tracker whose predictor imitates the motion in the training set. Tests show that the resulting trackers can be markedly tuned to desired curve shapes and classes of motions. Contents 1 Introduction 2 2 Tracking framework 2 2.1 Curve representation : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : 2 2.2 Tracking as estimation over time : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : 3 2.3 Rigid body transformations : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : 3 2.4 Curves in motion : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : 7 2.5 Discrete-time model : : : : : : : : : :...

In this paper we address the issue of recovering and segmenting the apparent velocity field in sequences of images. As for motion estimation, we minimize an objective function involving two robust terms. The first one cautiously captures the optical flow constraint, while the second (a priori) term incorporates a discontinuity-preserving smoothness constraint. To cope with the nonconvex minimization problem thus defined, we design an efficient deterministic multigrid procedure. It converges fast toward estimates of good quality, while revealing the large discontinuity structures of flow fields. We then propose an extension of the model by attaching to it a flexible object-based segmentation device based on deformable closed curves (different families of curve equipped with different kinds of prior can be easily supported). Experimental results on synthetic and natural sequences are presented, including an analysis of sensitivity to parameter tuning. INdex Terms--- Closed segmenting cu...

A new region-based approach to nonrigid motion tracking is described. Shape is defined in terms of a deformable triangular mesh that captures object shape plus a color texture map that captures object appearance. Photometric variations are also modeled. Nonrigid shape registration and motion tracking are achieved by posing the problem as an energy-based, robust minimization procedure. The approach provides robustness to occlusions, wrinkles, shadows, and specular highlights. The formulation is tailored to take advantage of texture mapping hardware available in many workstations, PC's, and game consoles. This enables nonrigid tracking at speeds approaching video rate. 1 Introduction A key open problem in tracking is that of encoding and comparing shapes as they undergo nonrigid deformation. Simply providing robustness to nonrigid deformation is insufficient, because deformation often provides important information about how shapes are related. To make things worse, tracking must also ...

Passive sensing of human hand and limb motion is important for a wide range of applications from human-computer interaction to athletic performance measurement. High degree of freedom articulated mechanisms like the human hand are difficult to track because of their large state space and complex image appearance. This article describes a model-based hand tracking system, called DigitEyes, that can recover the state of a 27 DOF hand model from gray scale images at speeds of up to 10 Hz. We employ kinematic and geometric hand models, along with a high temporal sampling rate, to decompose global image patterns into incremental, local motions of simple shapes. Hand pose and joint angles are estimated from line and point features extracted from images of unmarked, unadorned hands, taken from one or more viewpoints. We present some preliminary results on a 3D mouse interface based on the DigitEyes sensor. Contents 1 Introduction 2 2 The Articulated Mechanism Tracking Problem 2 3 State Mod...

Recent developments in video-tracking allow the outlines of moving, natural objects in a video-camera input stream to be tracked live, at full video-rate. Previous systems have been available to do this for specially illuminated objects or for naturally illuminated but polyhedral objects. Other systems have been able to track nonpolyhedral objects in motion, in some cases from live video, but following only centroids or key-points rather than tracking whole curves. The system described here can track accurately the curved silhouettes of moving non-polyhedral objects at frame-rate, for example hands, lips, legs, vehicles, fruit, and without any special hardware beyond a desktop workstation and a video-camera and framestore. The new algorithms are a synthesis of methods in deformable models, B-spline curve representation and control theory. This paper shows how such a facility can be used to turn parts of the body --- for instance, hands and lips --- into input devices. Rigid motion of a...

Computer sensing of hand and limb motion is an important problem for applications in HumanComputer Interaction (HCI), virtual reality, and athletic performance measurement. Commercially available sensors are invasive, and require the user to wear gloves or targets. We have developed a noninvasive vision-based hand tracking system, called DigitEyes. Employing a kinematic hand model, the DigitEyes system has demonstrated tracking performance at speeds of up to 10 Hz, using line and point features extracted from gray scale images of unadorned, unmarked hands. We describe an application of our sensor to a 3D mouse user-interface problem. 1 Introduction A "human sensor" capable of tracking a person's spatial motion using techniques from Computer Vision would be a powerful tool for human-computer interfaces. Such a sensor could be located in the user's environment (rather than on their person) and could operate under natural conditions of lighting and dress, providing a degree of convenien...

We present an approach for recovering surface shape from the occluding contour using an active (i.e., moving) observer. It is based onarelation between the geometries of a surface inascene and its occluding contour: If the viewing direction of the observer is along a principal direction for a surface point whose projection is on the contour, surface shape (i.e., curvature) at the surfacepoint can be recovered from the contour. Unlike previous approaches for recovering shape from the occluding contour, we use an observer that purposefully changes viewpoint in order to achieve a well-de ned geometric relationship with respect to a 3D shape prior to its recognition. We show that there is a simple and e cient viewing strategy that allows the observer to align the viewing direction with one of the two principal directions for a point on the surface. Experimental results demonstrate that our method can be easily implemented and can provide reliable shape information. 1

We develop a simple and very fast method for object tracking based exclusively on color information in digitized video images. Running on a Silicon Graphics R4600 Indy system with an IndyCam, our algorithm is capable of simultaneously tracking objects at full frame size (640 \Theta 480 pixels) and video frame rate (30 fps). Robustness with respect to occlusion is achieved via an explicit hypothesis-tree model of the occlusion process. We demonstrate the efficacy of our technique in the challenging task of tracking people, especially tracking human heads and hands. 1. Introduction A variety of problems of current interest in computer vision require the ability to track moving objects [2], whether for purposes of surveillance [9], manufacturing, video compression [6], visually "aware" information kiosks [19], etc. The fundamental challenges that drive much of the research in this field are the enormous data bandwidths implied by high resolution frames at high frame rates, a desire for ...

Existing object tracking algorithms generally use some form of local optimisation, assuming that an object 's position and shape change smoothly over time. In some situations this assumption is not valid: the trackable shape of an object may change discontinuously, for example if it is the 2D silhouette of a 3D object. In this paper we propose a novel method for modelling temporal shape discontinuities explicitly. Allowable shapes are represented as a union of (learned) bounded regions within a shape space. Discontinuous shape changes are described in terms of transitions between these regions. Transition probabilities are learned from training sequences and stored in a Markov model. In this way we can create `wormholes' in shape space. Tracking with such models is via an adaptation of the Condensation algorithm. 1 Introduction Models of shape have been used widely as an aid to tracking deformable objects [11, 2, 4, 1]. Existing tracking algorithms based on such models generally rel...