This paper presents a novel solution for flow-based tracking and 3D reconstruction of deforming objects in monocular image sequences. A non-rigid 3D object undergoing rotation and deformation can be effectively approximated using a linear combination of 3D basis shapes. This puts a bound on the rank of the tracking matrix. The rank constraint is used to achieve robust and precise low-level optical flow estimation without prior knowledge of the 3D shape of the object. The bound on the rank is also exploited to handle occlusion at the tracking level leading to the possibility of recovering the complete trajectories of occluded/disoccluded points. Following the same lowrank principle, the resulting flow matrix can be factored to get the 3D pose, configuration coefficients, and 3D basis shapes. The flow matrix is factored in an iterative manner, looping between solving for pose, configuration, and basis shapes. The flow-based tracking is applied to several video sequences and provides the input to the 3D non-rigid reconstruction task. Additional results on synthetic data and comparisons to ground truth complete the experiments.

Recovery of three diensWXzm (3D) sD) e and otion of non-sN;m[ s cenes fro a onocular videosdeomWW is i portant forapplications like robot navigation and hu an co puter interaction. If every point in thes cene rando ly oves it is i - posW=J= to recover the non-rigids-r es In practice, any non-rigid objects e.g. the hu an face under various expres[XFX] defor with certains tructures Theirs hapes can be regarded as a weighted co bination of certains hapebasXJ Shape and otion recovery unders uchs ituations has attracted uch interesX Previous work onthis proble [6, 4, 13] utilized only orthonor ality consWJNm ts on the ca era rotations (ro- tation constraints).This paper proves that usJ] only the rotation cons]N]m ts res]N] in a biguous and invalid smWWX];m[ The a biguity arisX fro the fact that thesmX e bas+ are not unique becaus their linear transJW ation is a news et of eligiblebasib To eli inate the a biguity, we propos as et of novel consNXNm ts basis constraints, which uniquely deter ine thesmW e bas;F We prove that, under the weak-p ers ective projection odel, enforcing both the bas= and the rotation consW+;m ts leads to a closNm[JF slosNm to the proble of non-rigids hape and otion recovery. The accuracy and robus;Wm[ of ourclos=;m[J slos=; is evaluated quantitatively on sm thetic data and qualitatively on real videoseomWN;JN 1

We present the design and implementation of a perceptual user interface for a responsive dialog-box agent that employs real-time computer vision to recognize user acknowledgements from head gestures (e.g., nod = yes). IBM Pupil-Cam technology together with anthropometric head and face measures are used to first detect the location of the user’s face. Salient facial features are then identified and tracked to compute the global 2-D motion direction of the head. For recognition, timings of natural gesture motion are incorporated into a state-space model. The interface is presented in the context of an enhanced text editor employing a perceptual dialog-box agent. 1.

This paper addresses the 3D tracking of pose and animation of the human face in monocular image sequences using active appearance models. The major problem of the classical appearance-based adaptation is the high computational time resulting from the inclusion of a synthesis step in the iterative optimization. Whenever the dimension of the face subspace is large, a real-time performance cannot be achieved. In this paper, we aim at designing a fast and stable active appearance model search for 3D face tracking. The main contribution is a search algorithm whose CPU-time is not dependent on the dimension of the face subspace. Using this algorithm, we show that both the CPU-time and the likelihood of a non accurate tracking are reduced. Experiments evaluating the effectiveness of the proposed algorithm are reported, as well as method comparison and tracking examples.

We present a novel fully automatic method for high resolution, non-rigid dense 3D point tracking. High quality dense point clouds of non-rigid geometry moving at video speeds are acquired using a phase-shifting structured light ranging technique. To use such data for the temporal study of subtle motions such as those seen in facial expressions, an efficient non-rigid 3D motion tracking algorithm is needed to establish inter-frame correspondences. The novelty of this paper is the development of an algorithmic framework for 3D tracking that unifies tracking of intensity and geometric features, using harmonic maps with added feature correspondence constraints. While the previous uses of harmonic maps provided only global alignment, the proposed introduction of interior feature constraints guarantees that non-rigid deformations will be accurately tracked as well. The harmonic map between two topological disks is a diffeomorphism with minimal stretching energy and bounded angle distortion. The map is stable, insensitive to resolution changes and is robust to noise. Due to the strong implicit and explicit smoothness constraints imposed by the algorithm and the high-resolution data, the resulting registration/deformation field is smooth, continuous and gives dense one-to-one inter-frame correspondences. Our method is validated through a series of experiments demonstrating its accuracy and efficiency. 1. Introduction and Previous

This paper describes a head-tracking algorithm that is based on recognition and correlation-based weighted interpolation. The input is a sequence of 3D depth images generated by a novel time-of-flight depth sensor. These are processed to segment the background and foreground, and the latter is used as the input to the head tracking algorithm, which is composed of three major modules: First, a depth signature is created out of the depth images. Next, the signature is compared against signatures that are collected in a training set of depth images. Finally, a correlation metric is calculated between most possible signature hits. The head location is calculated by interpolating among stored depth values, using the correlation metrics as the weights. This combination of depth sensing and recognition-based head tracking provides more than 90 percent success. Even if the track is temporarily lost, it is easily recovered when a good match is obtained from the training set. The use of depth images and recognition-based head tracking achieves robust real-time tracking results under extreme conditions such as 180-degree rotation, temporary occlusions, and complex backgrounds.

Abstract not found

Scene flow methods estimate the three-dimensional motion field for points in the world, using multi-camera video data. Such methods combine multi-view reconstruction with motion estimation. This paper describes an alternative formulation for dense scene flow estimation that provides reliable results using only two cameras by fusing stereo and optical flow estimation into a single coherent framework. Internally, the proposed algorithm generates probability distributions for optical flow and disparity. Taking into account the uncertainty in the intermediate stages allows for more reliable estimation of the 3D scene flow than previous methods allow. To handle the aperture problems inherent in the estimation of optical flow and disparity, a multi-scale method along with a novel region-based technique is used within a regularized solution. This combined approach both preserves discontinuities and prevents over-regularization – two problems commonly associated with the basic multi-scale approaches. Experiments with synthetic and real test data demonstrate the strength of the proposed approach.

The likelihood models used in probabilistic visual tracking applications are often complex non-linear and/or non-Gaussian functions, leading to analytically intractable inference. Solutions then require numerical approximation techniques, of which the particle filter is a popular choice. Particle filters, however, degrade in performance as the dimensionality of the state space increases and the support of the likelihood decreases. As an alternative to particle filters this paper introduces a variational approximation to the tracking recursion. The variational inference is intractable in itself, and is combined with an efficient importance sampling procedure to obtain the required estimates. The algorithm is shown to compare favourably with particle filtering techniques on a synthetic example and two real tracking problems. The first involves the tracking of a designated object in a video sequence based on its colour properties, whereas the second involves contour extraction in a single image. 1.

Recent advances in wearable sensing allow active control of the orientation of a body-mounted camera worn by a remote user. In this paper we consider the control of the active camera from head movements. In the context of teleoperation, these may be the head movements of a remote operator, perhaps acting as the wearer's assistant. The movements are likely to be larger than those in video-conference applications, and so frontal facial features are insufficient. The paper