Figure 1: Animation of a motion capture sequence taken for a subject, of whom we have a single body scan. The muscle deformations are synthesized automatically from the space of pose and body shape deformations. We introduce the SCAPE method (Shape Completion and Animation for PEople) — a data-driven method for building a human shape model that spans variation in both subject shape and pose. The method is based on a representation that incorporates both articulated and non-rigid deformations. We learn a pose deformation model that derives the non-rigid surface deformation as a function of the pose of the articulated skeleton. We also learn a separate model of variation based on body shape. Our two models can be combined to produce 3D surface models with realistic muscle deformation for different people in different poses, when neither appear in the training set. We show how the model can be used for shape completion — generating a complete surface mesh given a limited set of markers specifying the target shape. We present applications of shape completion to partial view completion and motion capture animation. In particular, our method is capable of constructing a high-quality animated surface model of a moving person, with realistic muscle deformation, using just a single static scan and a marker motion capture sequence of the person.

Details in mesh animations are difficult to generate but they have great impact on visual quality. In this work, we demonstrate a practical software system for capturing such details from multi-view video recordings. Given a stream of synchronized video images that record a human performance from multiple viewpoints and an articulated template of the performer, our system captures the motion of both the skeleton and the shape. The output mesh animation is enhanced with the details observed in the image silhouettes. For example, a performance in casual loose-fitting clothes will generate mesh animations with flowing garment motions. We accomplish this with a fast pose tracking method followed by nonrigid deformation of the template to fit the silhouettes. The entire process takes less than sixteen seconds per frame and requires no markers or texture cues. Captured meshes are in full correspondence making them readily usable for editing operations including texturing, deformation transfer, and deformation model learning.

Shape-From-Silhouette (SFS) is a shape reconstruction method which constructs a 3D shape estimate of an object using silhouette images of the object. The output of a SFS algorithm is known as the Visual Hull (VH). Traditionally SFS is either performed on static objects, or separately at each time instant in the case of videos of moving objects. In this paper we develop a theory of performing SFS across time: estimating the shape of a dynamic object (with unknown motion) by combining all of the silhouette images of the object over time. We first introduce a one dimensional element called a Bounding Edge to represent the Visual Hull. We then show that aligning two Visual Hulls using just their silhouettes is in general ambiguous and derive the geometric constraints (in terms of Bounding Edges) that govern the alignment. To break the alignment ambiguity, we combine stereo information with silhouette information and derive a Temporal SFS algorithm which consists of two steps: (1) estimate the motion of the objects over time (Visual Hull Alignment) and (2) combine the silhouette information using the estimated motion (Visual Hull Refinement). The algorithm is first developed for rigid objects and then extended to articulated objects. In the Part II of this paper we apply our temporal SFS algorithm to two human-related applications: (1) the acquisition of detailed human kinematic models and (2) marker-less motion tracking.

We introduce an efficient compression method for animated sequences of irregular meshes of the same connectivity. Our approach is to transform the original input meshes with an anisotropic wavelet transform running on top of a progressive mesh hierarchy, and progressively encode the resulting wavelet details. For temporally coherent mesh sequences we get additional improvement by encoding the differences of the wavelet coefficients. The resulting compression scheme is scalable, efficient, and significantly improves upon the current state of the art for the animated mesh compression.

We present a novel algorithm to jointly capture the motion and the dynamic shape of humans from multiple video streams without using optical markers. Instead of relying on kinematic skeletons, as traditional motion capture methods, our approach uses a deformable high-quality mesh of a human as scene representation. It jointly uses an imagebased 3D correspondence estimation algorithm and a fast Laplacian mesh deformation scheme to capture both motion and surface deformation of the actor from the input video footage. As opposed to many related methods, our algorithm can track people wearing wide apparel, it can straightforwardly be applied to any type of subject, e.g. animals, and it preserves the connectivity of the mesh over time. We demonstrate the performance of our approach using synthetic and captured real-world video sequences and validate its accuracy by comparison to the ground truth. 1.

video In model-based free-viewpoint video, a detailed representation of the time-varying geometry of a real-word scene is used to generate renditions of it from novel viewpoints. In this paper, we present a method for reconstructing such a dynamic geometry model of a human actor from multi-view video. In a two-step procedure, first the spatio-temporally consistent shape and poses of a generic human body model are estimated by means of a silhouette-based analysis-by-synthesis method. In a second step, subtle details in surface geometry that are specific to each particular time step are recovered by enforcing a color-consistency criterion. By this means, we generate a realistic representation of the time-varying geometry of a moving person that also reproduces these dynamic surface variations. 1

Compression of 3D mesh animations is a topic that has received increased attention in recent years, due to increasing capabilities of modern processing and displaying hardware. In this paper we present an improved approach based on known techniques, such as principal component analysis (PCA) and EdgeBreaker, which allows efficient encoding of highly detailed dynamic meshes, exploiting both spatial and temporal coherence. We present the results of our method compared with similar approaches described in literature, showing that using our approach we can achieve better performance in terms of rate/distortion ratio. Index Terms — Dynamic mesh, compression, PCA, EdgeBreaker, coherency exploitation, entropy

We present a new performance capture approach that incorporates a physically-based cloth model to reconstruct a rigged fullyanimatable virtual double of a real person in loose apparel from multi-view video recordings. Our algorithm only requires a minimum of manual interaction. Without the use of optical markers in the scene, our algorithm first reconstructs skeleton motion and detailed time-varying surface geometry of a real person from a reference video sequence. These captured reference performance data are then analyzed to automatically identify non-rigidly deforming pieces of apparel on the animated geometry. For each piece of apparel, parameters of a physically-based real-time cloth simulation model are estimated, and surface geometry of occluded body regions is approximated. The reconstructed character model comprises a skeleton-based representation for the actual body parts and a physically-based simulation model for the apparel. In contrast to previous performance capture methods, we can now also create new real-time animations of actors captured in general apparel.

Abstract not found

We present an automatic approach for calibrating a network of cameras using live video captured from them. Our method requires video sequences containing moving people or objects but does not require any special calibration data. The silhouettes of these moving objects visible in a pair of views, are used to compute the epipolar geometry of that camera pair. The fundamental matrices computed by this method are used to first obtain a projective reconstruction of the complete camera configuration. Self-calibration is then used to upgrade the projective reconstruction into a metric reconstruction. We have extended our approach to deal with unsynchronized video sequences captured at the same frame-rate, by simultaneously recovering the epipolar geometry as well as the temporal offset between a pair of cameras. We use our approach to calibrate and synchronize a four-camera system using archived video containing a moving person. Next, the silhouettes are used to construct the visual hull of the moving person using known Shape-from-Silhouette algorithms. Additional experiments on computing the fundamental matrix of two views from silhouettes are also performed. 1.