We propose a new algorithm for dense optical flow computation. Dense optical flow schemes are challenged by the presence of motion discontinuities. In state of the art optical flow methods, over-smoothing of flow discontinuities accounts for most of the error. A breakthrough in the performance of optical flow computation has recently been achieved by Brox et al. Our algorithm embeds their functional within a two phase active contour segmentation framework. Piecewise-smooth flow fields are accommodated and flow boundaries are crisp. Experimental results show the superiority of our algorithm with respect to alternative techniques. We also study a special case of optical flow computation, in which the camera is static. In this case we utilize a known background image to separate the moving elements in the sequence from the static elements. Tests with challenging real world sequences demonstrate the performance gains made possible by incorporating the static camera assumption in our algorithm. 1

We present an incremental approach to motion segmentation. Feature points are detected and tracked throughout an image sequence, and the features are grouped using a region-growing algorithm with an affine motion model. The primary parameter used by the algorithm is the amount of evidence that must accumulate before features are grouped. Contrasted with previous work, the algorithm allows for a variable number of image frames to affect the decision process, thus enabling objects to be detected independently of their velocity in the image. Procedures are presented for grouping features, measuring the consistency of the resulting groups, assimilating new features into existing groups, and splitting groups over time. Experimental results on a number of challenging image sequences demonstrate the effectiveness of the technique. 1

Abstract. We use a simple yet powerful higher-order conditional random field (CRF) to model optical flow. It consists of a standard photoconsistency cost and a prior on affine motions both modeled in terms of higher-order potential functions. Reasoning jointly over a large set of unknown variables provides more reliable motion estimates and a robust matching criterion. One of the main contributions is that unlike previous region-based methods, we omit the assumption of constant flow. Instead, we consider local affine warps whose likelihood energy can be computed exactly without approximations. This results in a tractable, so-called, higher-order likelihood function. We realize this idea by employing triangulation meshes which immensely reduce the complexity of the problem. Optimization is performed by hierarchical QPBO moves and an adaptive mesh refinement strategy. Experiments show that we achieve high-quality motion fields on several data sets including the Middlebury optical flow database. 1

Abstract—We present a real-time incremental approach to motion segmentation operating on sparse feature points. In contrast to previous work, the algorithm allows for a variable number of image frames to affect the segmentation process, thus enabling an arbitrary number of objects traveling at different relative speeds to be detected. Feature points are detected and tracked throughout an image sequence, and the features are grouped using a spatially constrained expectation–maximization (EM) algorithm that models the interactions between neighboring features using the Markov assumption. The primary parameter used by the algorithm is the amount of evidence that must accumulate before features are grouped. A statistical goodness-of-fit test monitors the change in the motion parameters of a group over time in order to automatically update the reference frame. Experimental results on a number of challenging image sequences demonstrate the effectiveness and computational efficiency of the technique. Index Terms—Expectation–maximization (EM), feature tracking, motion segmentation. I.

Abstract. Motion estimation is usually based on the brightness constancy assumption. This assumption holds well for rigid objects with a Lambertian surface, but it is less appropriate for fluid and gaseous materials. For these materials a variant of this assumption, which we call the brightness conservation assumption should be employed. Under this assumption an object’s brightness can diffuse to its neighborhood. We propose a method for detecting regions of dynamic texture in image sequences. Segmentation into regions of static and dynamic texture is achieved by using a level set scheme. The level set function separates the images into areas obeying brightness constancy and those which obey brightness conservation. Experimental results on challenging image sequences demonstrate the success of the segmentation scheme and validate the model.

Abstract. The estimation of optical flow fields from image sequences is incorporated in a Mumford–Shah approach for image denoising and edge detection. Possibly noisy image sequences are considered as input and a piecewise smooth image intensity, a piecewise smooth motion field, and a joint discontinuity set are obtained as minimizers of the functional. The method simultaneously detects image edges and motion field discontinuities in a rigorous and robust way. It is able to handle information on motion that is concentrated on edges. Inherent to it is a natural multiscale approximation that is closely related to the phase field approximation for edge detection by Ambrosio and Tortorelli. We present an implementation for two-dimensional image sequences with finite elements in space and time. This leads to three linear systems of equations, which have to be solved in a suitable iterative minimization procedure. Numerical results and different applications underline the robustness of the approach presented.

I propose a notion of visual information as the complexity not of the raw images, but of the images after the effects of nuisance factors such as viewpoint and illumination are discounted. It is rooted in ideas of J. J. Gibson, and stands in contrast to traditional information as entropy or coding length of the data regardless of its use, and regardless of the nuisance factors affecting it. The non-invertibility of nuisances such as occlusion and quantization induces an “information gap ” that can only be bridged by controlling the data acquisition process. Measuring visual information entails early vision operations, tailored to the structure of the nuisances so as to be “lossless ” with respect to visual decision and control tasks (as opposed to data transmission and storage tasks implicit in traditional Information Theory). I illustrate these ideas on visual exploration, whereby a “Shannonian Explorer ” guided by the entropy of the data navigates unaware of the structure of the physical space surrounding it, while a “Gibsonian Explorer ” is guided by the topology of the environment, despite measuring only images of it, without performing 3D reconstruction. The operational definition of visual information suggests desirable properties that a visual representation should possess to best accomplish vision-based decision and control tasks. 1.

Abstract. The detection of motion boundaries has been and remains a longstanding challenge in computer vision. In this paper, the recovery of motion boundaries is recast in a broader scope, as focus is placed on the more general problem of detecting spacetime structure boundaries, where motion boundaries constitute a special case. This recasting allows uniform consideration of boundaries between a wider class of spacetime patterns than previously considered in the literature, both coherent motion as well as additional dynamic patterns. Examples of dynamic patterns beyond standard motion that are encompassed by the proposed approach include, flicker, transparency and various dynamic textures (e.g., scintillation). Toward this end, a novel representation and method for detecting these boundaries in raw image sequence data are presented. Central to the representation is the description of oriented spacetime structure in a distributed manner. Empirical evaluation of the proposed boundary detector on challenging natural imagery suggests its efficacy. 1

We present a motion segmentation algorithm that partitions the image plane into disjoint regions based on their parametric motion. It relies on a finer partitioning of the image domain into regions of uniform photometric properties, with motion segments made of unions of such “superpixels.” We exploit recent advances in combinatorial graph optimization that yield computationally efficient estimates. The energy functional is built on a superpixel graph, and is iteratively minimized by computing a parametric motion model in closed-form, followed by a graph cut of the superpixel adjacency graph. It generalizes naturally to multilabel partitions that can handle multiple motions. 1.

We define a novel metric on the space of closed planar curves. According to this metric centroid translations, scale changes and deformations are orthogonal, and the metric is also invariant with respect to reparameterizations of the curve. The Riemannian structure that is induced on the space of curves is a smooth Riemannian manifold, which is isometric to a classical well-known manifold. As a consequence, geodesics and gradients of energies defined on the space can be computed using fast closed-form formulas, and this has obvious benefits in numerical applications. The obtained Riemannian manifold of curves is apt to address complex problems in computer vision; one such example is the tracking of highly deforming objects. Previous works have assumed that the object deformation is smooth, which is realistic for the tracking problem, but most have restricted the deformation to belong to a finite-dimensional group – such as affine motions – or to finitely-parameterized models. This is too restrictive for highly deforming objects such as the contour of a beating heart. We adopt the smoothness assumption implicit in previous work, but we lift the restriction to finite-dimensional motions/deformations. We define a dynamical model in this Riemannian manifold of curves, and use it to perform filtering and prediction to infer and extrapolate not just the pose (a finitely parameterized quantity) of an object, but its deformation (an infinite-dimensional quantity) as well. We illustrate these ideas using a simple first-order dynamical model, and show that it can be effective even on data sets where existing methods fail. 1