While image registration has been studied in different areas of computer vision, aligning images depicting different scenes remains a challenging problem, closer to recognition than to image matching. Analogous to optical flow, where an image is aligned to its temporally adjacent frame, we propose SIFT flow, a method to align an image to its neighbors in a large image collection consisting of a variety of scenes. For a query image, histogram intersection on a bag-of-visual-words representation is used to find the set of nearest neighbors in the database. The SIFT flow algorithm then consists of matching densely sampled SIFT features between the two images, while preserving spatial discontinuities. The use of SIFT features allows robust matching across different scene/object appearances and the discontinuity-preserving spatial model allows matching of objects located at different parts of the scene. Experiments show that the proposed approach is able to robustly align complicated scenes with large spatial distortions. We collect a large database of videos and apply the SIFT flow algorithm to two applications: (i) motion field prediction from a single static image and (ii) motion synthesis via transfer of moving objects.

In this paper we propose a novel nonparametric approach for object recognition and scene parsing using dense scene alignment. Given an input image, we retrieve its best matches from a large database with annotated images using our modified, coarse-to-fine SIFT flow algorithm that aligns the structures within two images. Based on the dense scene correspondence obtained from the SIFT flow, our system warps the existing annotations, and integrates multiple cues in a Markov random field framework to segment and recognize the query image. Promising experimental results have been achieved by our nonparametric scene parsing system on a challenging database. Compared to existing object recognition approaches that require training for each object category, our system is easy to implement, has few parameters, and embeds contextual information naturally in the retrieval/alignment procedure. 1.

Accurate estimation of optical flow is a challenging task, which often requires addressing difficult energy optimization problems. To solve them, most top-performing methods rely on continuous optimization algorithms. The modeling accuracy of the energy in this case is often traded for its tractability. This is in contrast to the related problem of narrow-baseline stereo matching, where the top-performing methods employ powerful discrete optimization algorithms such as graph cuts and message-passing to optimize highly non-convex energies. In this paper, we demonstrate how similar non-convex energies can be formulated and optimized discretely in the context of optical flow estimation. Starting with a set of candidate solutions that are produced by fast continuous flow estimation algorithms, the proposed method iteratively fuses these candidate solutions by the computation of minimum cuts on graphs. The obtained continuous-valued fusion result is then further improved using local gradient descent. Experimentally, we demonstrate that the proposed energy is an accurate model and that the proposed discretecontinuous optimization scheme not only finds lower energy solutions than traditional discrete or continuous optimization techniques, but also leads to flow estimates that outperform the current state-of-the-art.

We consider the task of obtaining the maximum a posteriori estimate of discrete pairwise random fields with arbitrary unary potentials and semimetric pairwise potentials. For this problem, we propose an accurate hierarchical move making strategy where each move is computed efficiently by solving an st-MINCUT problem. Unlike previous move making approaches, e.g. the widely used α-expansion algorithm, our method obtains the guarantees of the standard linear programming (LP) relaxation for the important special case of metric labeling. Unlike the existing LP relaxation solvers, e.g. interior-point algorithms or tree-reweighted message passing, our method is significantly faster as it uses only the efficient st-MINCUT algorithm in its design. Using both synthetic and real data experiments, we show that our technique outperforms several commonly used algorithms. 1

While image alignment has been studied in different areas of computer vision for decades, aligning images depicting different scenes remains a challenging problem. Analogous to optical flow where an image is aligned to its temporally adjacent frame, we propose SIFT flow, a method to align an image to its nearest neighbors in a large image corpus containing a variety of scenes. The SIFT flow algorithm consists of matching densely sampled, pixel-wise SIFT features between two images, while preserving spatial discontinuities. The SIFT features allow robust matching across different scene/object appearances, whereas the discontinuitypreserving spatial model allows matching of objects located at different parts of the scene. Experiments show that the proposed approach robustly aligns complex scene pairs containing significant spatial differences. Based on SIFT flow, we propose an alignmentbased large database framework for image analysis and synthesis, where image information is transferred from the nearest neighbors to a query image according to the dense scene correspondence. This framework is demonstrated through concrete applications, such as motion field prediction from a single image, motion synthesis via object transfer, satellite image registration and face recognition.

source target our approach [Papenberg et al. 2007] [Glocker et al. 2008] Figure 1: Compared with the state-of-the-art in deformable image registration, our novel approach reaches plausible results even for challenging configurations undergoing large amounts of free-form deformation and notable changes in appearance. We present a new approach to deformable image registration suitable for articulated images such as hand-drawn cartoon characters and human postures. For such type of data state-of-the-art techniques typically yield undesirable results. We propose a novel geometrically motivated iterative scheme where point movements are decoupled from shape consistency. By combining locally optimal block matching with as-rigid-as-possible shape regularization, our algorithm allows us to register images undergoing large free-form deformations and appearance variations. We demonstrate its practical usability in various challenging tasks performed in the cartoon animation production pipeline including unsupervised inbetweening, example-based shape deformation, auto-painting, editing, and motion retargeting.

An efficient registration algorithm working on non-rigid 3D objects is presented. We formulate the registration as a discrete labeling problem on MRFs model whose energy can be minimized by optimization techniques in the literature. Due to the huge search range in three-dimensional space, previous approaches produces a vast amount of labels for a node in the MRFs graph. To reduce the number of labels, we decompose a node into three nodes so that the labels in each node represent just one-dimensional displacement. This procedure introduces a factor node with a clique potential of size three, defining ternary interaction between the decomposed nodes. We convert the factor node into pairwise interactions and adopt the tree-reweighted message passing technique, which guarantees the convergence of lower bound of the energy function. In experiments we use clinical and synthetically deformed 3D medical images. Result shows the proposed method enhances computational efficiency without loss of accuracy. 1

Abstract. Labeling of discrete Markov Random Fields (MRFs) has become an attractive approach for solving the problem of non-rigid image registration. Here, regularization plays an important role in order to obtain smooth deformations for the inherent ill-posed problem. Smoothness is achieved by penalizing the derivatives of the displacement field. However, efficient optimization strategies (based on iterative graph-cuts) are only available for first-order MRFs which contain cliques of size up to two. Higher-order cliques require graph modifications and insertion of auxiliary nodes, while pairwise interactions actually allow only regularization based on the first-order derivatives. In this paper, we propose an approximated curvature penalty using second-order derivatives defined on the MRF pairwise potentials. In our experiments, we demonstrate that our approximated term has similar properties as higher-order approaches (invariance to linear transformations), while the computational efficiency of pairwise models is preserved. 1

Abstract. We propose a reliability measure that identifies informative image cues useful for registration, and present a novel, data-driven approach to spatially adapt regularization to the local image content via use of the proposed measure. We illustrate the generality of this adaptive regularization approach within a powerful discrete optimization framework and present various ways to construct a spatially varying regularization weight based on the proposed measure. We evaluate our approach within the registration process using synthetic experiments and demonstrate its utility in real applications. As our results demonstrate, our approach yielded higher registration accuracy than non-adaptive approaches and the proposed reliability measure performed robustly even in the presences of noise and intensity inhomogenity. 1

Abstract. This paper describes a novel approach to recovering a parametric deformation that optimally registers one image to another. The method proceeds by constructing a global convex approximation to the match function which can be optimized using interior point methods. The paper also describes how one can exploit the structure of the resulting optimization problem to develop efficient and effective matching algorithms. Results obtained by applying the proposed scheme to a variety of images are presented. 1