This paper studies the problem of sequence-to-sequence alignment, namely establishing correspondences in time and in space between two di erent video sequences of the same dynamic scene. The sequences are recorded by uncalibrated video cameras, which are either stationary or jointly moving, with xed (but unknown) internal parameters and relative inter-camera external parameters. Temporal variations between image frames (such as moving objects or changes in scene illumination) are powerful cues for alignment, which cannot be exploited by standard image-toimage alignment techniques. We show that by folding spatial and temporal cues into a single alignment framework, situations which are inherently ambiguous for traditional image-to-image alignment methods, are often uniquely resolved by sequence-to-sequence alignment. Furthermore, the ability to align and integrate information across multiple video sequences both in time and in space gives rise to new video applications that are not possible when only image-to-image alignment is used. 1

For stereopsis, images of a given scene must be captured at the same instant to ensure temporal consistency. For sequences of images (i.e. video streams) this requires the potentially costly and technically complex process of synchronizing cameras. We present a simple but effective method for automatically recovering the sub-frame temporal offset between image sequences taken using unsynchronized cameras. Having recovered the offset, we obtain the affine structure of a non-rigid motion. The technique is demonstrated for the application of human motion capture. 1

3D reconstruction of a dynamic non-rigid scene from features in two cameras usually requires synchronization and correspondences between the cameras. These may be hard to achieve due to occlusions, wide base-line, different zoom scales, etc. In this work we present an algorithm for reconstructing a dynamic scene from sequences acquired by two uncalibrated non-synchronized fixed affine cameras. It is assumed that (possibly) different points are tracked in the two sequences. The only con-straint used to relate the two cameras is that every 3D point tracked in one sequence can be described as a linear combination of some of the 3D points tracked in the other sequence. Such constraint is useful, for example, for articulated objects. We may track some points on an arm in the first sequence, and some other points on the same arm in the second sequence. On the other extreme, this model can be used for generally moving points tracked in both sequences without knowing the correct permutation. In between, this model can cover non-rigid bodies, with local rigidity constraints. 1 We present linear algorithms for synchronizing the two sequences and reconstructing the 3D points tracked in both views. Outlier points are automatically detected and discarded. The algo-rithm can handle both 3D objects and planar objects in a unified framework, therefore avoiding numerical problems existing in other methods. 1

Abstract. We introduced an algorithm for sequence alignment, based on maximizing local space-time correlations. Our algorithm aligns sequences of the same action performed at different times and places by different people, possibly at different speeds, and wearing different clothes. Moreover, the algorithm offers a unified approach to the problem of sequence alignment for a wide range of scenarios (e.g., sequence pairs taken with stationary or jointly moving cameras, with the same or different photometric properties, with or without moving objects). Our algorithm is applied directly to the dense space-time intensity information of the two sequences (or to filtered versions of them). This is done without prior segmentation of foreground moving objects, and without prior detection of corresponding features across the sequences. Examples of challenging sequences with complex actions are shown, including ballet dancing, actions in the presence of other complex scene dynamics (clutter), as well as multi-sensor sequence pairs. 1

Human motion capture typically requires several high quality, synchronized and calibrated cameras in a studio environment and can be potentially costly and technically complex. Instead, we propose a system which combines and improves upon two existing techniques, yielding an efficient method that recovers maximum likelihood joint angles and anthropomorphic data of the subject by factorization. The first technique concerns using a rank constraint framework to synchronize sequences of non-rigid motions where we extend affine methods to perspective and homography projection models. The second is a self-calibration method for two affine cameras, using constraints derived from prior knowledge of the underlying structure. We propose a minimal parameterization of the system to obtain an initial solution then apply a full bundle adjustment over the free parameters based on a geometric error. We demonstrate the efficacy of our method by comparing the recovered structure and motion with that from a commercial motion capture system.

Recovery of dynamic scene properties from multiple videos usually requires the manipulation of synchronous (simultaneously captured) frames. This paper is concerned with the automated determination of this synchrony when the temporal alignment of sequences is unknown. A cost function characterising departure from synchrony is first evolved for the case in which two videos are generated by cameras that may be moving. A novel voting method is then presented for minimising the cost function in the case where the ratio of the cameras ’ frame rates is unknown. Experimental results indicate this relatively general approach holds promise. 1.

Abstract. A simple, stable and generic approach for estimation of relative positions and orientations of multiple rigidly coupled cameras is presented in this paper. The algorithm does not impose constraints on the field of view of the cameras and works even in the extreme case when the sequences from the different cameras are totally disjoint (i.e. when no part of the scene is captured by more than one camera). The influence of the rig motion on the existence of a unique solution is investigated and degenerate rig motions are identified. Each camera captures an individual sequence which is afterwards processed by a structure and motion (SAM) algorithm resulting in positions and orientations for each camera. The unknown relative transformations between the rigidly coupled cameras are estimated utilizing the rigidity constraint of the rig. 1

So far extending light field rendering to dynamic scenes has been trivially treated as the rendering of static light fields stacked in time. This type of approaches requires input video sequences in strict synchronization and allows only discrete exploration in the temporal domain determined by the capture rate. In this paper we propose a novel framework, space-time light field rendering, which allows continuous exploration of a dynamic scene in both spatial and temporal domain with unsynchronized input video sequences. In order to synthesize novel views from any viewpoint at any time instant, we develop a two-stage rendering algorithm. We first interpolate in the temporal domain to generate globally synchronized images using a robust spatial-temporal image registration algorithm followed by edge-preserving image morphing. We then interpolate those software-synchronized images in the spatial domain to synthesize the final view. Our experimental results show that our approach is robust and capable of maintaining photo-realistic results.

Over the past decade, tremendous amount of research activity has focused around the problem of localization in GPS denied environments. Challenges with localization are highlighted in human wearable systems where the operator can freely move through both indoors and outdoors. In this paper, we present a robust method that addresses these challenges using a human wearable system with two pairs of backward and forward looking stereo cameras together with an inertial measurement unit (IMU). This algorithm can run in real-time with 15Hz update rate on a dual-core 2GHz laptop PC and it is designed to be a highly accurate local (relative) pose estimation mechanism acting as the front-end to a Simultaneous Localization and Mapping (SLAM) type method capable of global corrections through landmark matching. Extensive tests of our prototype system so far, reveal that without any global landmark matching, we achieve between 0.5 % and 1 % accuracy in localizing a person over a 500 meter travel indoors and outdoors. To our knowledge, such performance results with a real time system have not been reported before. 1.

We propose a system for robust modeling and visualisation of complex outdoor scenes from multi-camera image sequences and additional sensor information. A camera rig with one or more fire-wire cameras is used in conjunction with a 3-axis rotation sensor to robustly obtain a calibration of the scene with an uncalibrated structure from motion approach. Dense depth maps are estimated with multi-viewpoint stereo and the scene is visualized from a plenoptic representation of the scene.