Abstract. Since their introduction as a means of front propagation and their first application to edge-based segmentation in the early 90’s, level set methods have become increasingly popular as a general framework for image segmentation. In this paper, we present a survey of a specific class of region-based level set segmentation methods and clarify how they can all be derived from a common statistical framework. Region-based segmentation schemes aim at partitioning the image domain by progressively fitting statistical models to the intensity, color, texture or motion in each of a set of regions. In contrast to edge-based schemes such as the classical Snakes, region-based methods tend to be less sensitive to noise. For typical images, the respective cost functionals tend to have less local minima which makes them particularly well-suited for local optimization methods such as the level set method. We detail a general statistical formulation for level set segmentation. Subsequently, we clarify how the integration of various low level criteria leads to a set of cost functionals and point out relations between the different segmentation schemes. In experimental results, we demonstrate how the level set function is driven to partition the image plane into domains of coherent color, texture, dynamic texture or motion. Moreover, the Bayesian formulation allows to introduce prior shape knowledge into the level set method. We briefly review a number of advances in this domain.

Abstract. We present a novel variational approach for segmenting the image plane into a set of regions of parametric motion on the basis of two consecutive frames from an image sequence. Our model is based on a conditional probability for the spatio-temporal image gradient, given a particular velocity model, and on a geometric prior on the estimated motion field favoring motion boundaries of minimal length. Exploiting the Bayesian framework, we derive a cost functional which depends on parametric motion models for each of a set of regions and on the boundary separating these regions. The resulting functional can be interpreted as an extension of the Mumford-Shah functional from intensity segmentation to motion segmentation. In contrast to most alternative approaches, the problems of segmentation and motion estimation are jointly solved by continuous minimization of a single functional. Minimizing this functional with respect to its dynamic variables results in an eigenvalue problem for the motion parameters and in a gradient descent evolution for the motion discontinuity set. We propose two different representations of this motion boundary: an explicit spline-based implementation which can be applied to the motion-based tracking of a single moving object, and an implicit multiphase level set implementation which allows for the segmentation of an arbitrary number of multiply connected moving objects. Numerical results both for simulated ground truth experiments and for real-world sequences demonstrate the capacity of our approach to segment objects based exclusively on their relative motion.

Abstract. In this paper we present a new technique to extract layers in a video sequence. To this end, we assume that the observed scene is composed of several transparent layers, that their motion in the 2D plane can be approximated with an affine model. The objective of our approach is the estimation of these motion models as well as the estimation of their support in the image domain. Our technique is based on an iterative process that integrates robust motion estimation, MRF-based formulation, combinatorial optimization and the use of visual as well as motion features to recover the parameters of the motion models as well as their support layers. Special handling of occlusions as well as adaptive techniques to detect new objects in the scene are also considered. Promising results demonstrate the potentials of our approach. 1

Spatiotemporal data is associated with vast amounts of raw samples. Given the limited computational resources typically available, an initial organization of this data supporting semantically meaningful lines of inquiry would facilitate efficient processing. In this paper, a new representation for grouping raw image data into a set of coherent spacetime regions is proposed. Unique in this proposal is that coherency is related to a richer description of local spacetime structure than generally considered. In particular, the representation describes the presence of particular oriented spacetime structures in a distributed manner. A key advantage of this representation is its ability to signal the presence of multiple oriented structures at a given spacetime location. More generally, the abstraction allows for the description and grouping of motion and non-motionrelated patterns in a uniform manner. Empirical evaluation of the grouping method on synthetic and challenging natural imagery suggests its efficacy.

This paper describes a novel method for performing spatially coherent motion estimation by integrating region and boundary information. The method begins with a layered, parametric flow model. Since the resulting flow estimates are typically sparse, we use the computed motion in a novel way to compare intensity values between images, thereby providing improved spatial coherence of a moving region. This dense set of intensity constraints is then used to initialize an active contour, which is influenced by both motion and intensity data to track the object's boundary.

Abstract. We present a variational approach for segmenting the image plane into regions of piecewise parametric motion given two or more frames from an image sequence. Our model is based on a conditional probability for the spatio-temporal image gradient, given a particular velocity model, and on a geometric prior on the estimated motion field favoring motion boundaries of minimal length. We cast the problem of motion segmentation as one of Bayesian inference, we derive a cost functional which depends on parametric motion models for each of a set of domains and on the boundary separating them. The resulting functional can be interpreted as an extension of the Mumford-Shah functional from intensity segmentation to motion segmentation. In contrast to most alternative approaches, the problems of segmentation and motion estimation are jointly solved by continuous minimization of a single functional. Minimization results in an eigenvalue problem for the motion parameters and in a gradient descent evolution for the motion boundary. The evolution of the motion boundaries is implemented by a multiphase level set formulation which allows for the segmentation of an arbitrary number of multiply connected moving objects. We further extend this approach to the segmentation of space-time volumes of coherent motion from video sequences. To this end, motion boundaries are represented by a set of surfaces in space-time. An implementation by a higher-dimensional multiphase level set model allows the evolving surfaces to undergo topological changes. In contrast to an iterative segmentation of consecutive frame pairs, a constraint on the area of these surfaces leads to an additional temporal regularization of the computed motion boundaries. Numerical results demonstrate the capacity of our approach to segment objects based exclusively on their relative motion. 1

In this paper we present a generative model for image sequences, which can be applied to motion segmentation and tracking, and to image sequence compression. The model consists of regions of relatively constant color that have a motion model explaining their motion in time. At each frame, the model can allow accretion and deletion of pixels. We also present an algorithm for maximizing the posterior probability of the image sequence model, based on the recently introduced Swendsen-Wang Cuts algorithm. We show how one can use multiple cues and model switching in a reversible manner to make better bottom-up proposals. The algorithm works on the 3d spatiotemporal pixel volume to reassign entire trajectories of constant color in very few steps, while maintaining detailed balance.

We present a new algorithm that does motion segmentation by tracking small textured patches and then clustering them using EM. A small patch has the advantage that its motion is well modeled by uniform flow and runs a lower risk of boundary inclusion. Inherently, a small patch has less data so it is more susceptible to noise and it is not well suited to fit locally higher order flow models. To overcome these difficulties, we introduce a motion coherence detector to select only the best features and an efficient statistical technique to compute segment-wise affine flow from the EM clustering parameters. We incorporate a residual noise model without any statistical independence assumption and an efficient # test for the noise model to obtain dense segmentation. Computational efficiency is striven for within a rigorous mathematical framework. Experiments with real image sequences show good segments under a variety of conditions.

In this paper, several important issues related to visual motion analysis are addressed with a focus on the type of motion information to be estimated and the way contextual information is expressed and exploited. Assumptions (i.e., data models) must be formulated to relate the observed image intensities with motion, and other constraints (i.e., motion models) must be added to solve problems like motion segmentation, optical flow computation, or motion recognition. The motion models are supposed to capture known, expected or learned properties of the motion field, and this implies to somehow introduce spatial coherence or more generally contextual information. The latter can be formalized in a probabilistic way with local conditional densities as in Markov models. It can also rely on predefined spatial supports (e.g., blocks or pre-segmented regions). The classic mathematical expressions associated with the visual motion information are of two types. Some are continuous variables to represent velocity vectors or parametric motion models. The other are discrete variables or symbolic labels to code motion detection output (binary labels) or motion segmentation output (numbers of the motion regions or layers). We introduce new models, called mixed-state auto-models, whose variables belong to a domain formed by the union of discrete and continuous values, and which include local spatial contextual information. We describe how such...

Spatiotemporal image data sets, like 4D CT or dynamic MRI, open up the possibility to estimate respiratory induced tumor and organ motion and to generate four-dimensional models that describe the temporal change in position and shape of structures of interest. However, two main problems arise: the structures of interest have to be segmented in the 4D data set and and the organ motion has to be estimated in the temporal image sequence. This paper presents a variational approach for simultaneous segmentation and registration applied to temporal image sequences. The proposed method assumes a known segmentation in one frame and then recovers nonlinear registration and segmentation in other frames by minimizing a cost function that combines intensity-based registration, level-set segmentation as well as prior shape and intensity knowledge. The purpose of the presented method is to estimate respiration induced organ motion in spatiotemporal CT image sequences and to segment a structure of interest simultaneously. A validation of the combined registration and segmentation approach is presented using low dose 4D CT data sets of the liver. The results demonstrate that the simultaneous solution of both problems improves the segmentation performance over a sequential application of the registration and segmentation steps.