## Motion Segmentation and Tracking Using Normalized Cuts (1998)

### Cached

### Download Links

- [www.cs.berkeley.edu]
- [www.cs.berkeley.edu]
- [www.cs.cmu.edu]
- DBLP

### Other Repositories/Bibliography

Citations: | 146 - 5 self |

### BibTeX

@MISC{Shi98motionsegmentation,

author = {Jianbo Shi and Jitendra Malik},

title = {Motion Segmentation and Tracking Using Normalized Cuts},

year = {1998}

}

### Years of Citing Articles

### OpenURL

### Abstract

We propose a motion segmentation algorithm that aims to break a scene into its most prominent moving groups. A weighted graph is constructed on the ira. age sequence by connecting pixels that arc in the spatio-temporal neighborhood of each other. At each pizel, we define motion profile vectors which capture the probability distribution of the image veloczty. The distance between motion profiles is used to assign a weight on the graph edges. 5rsmg normalized cuts we find the most salient partitions of the spatiotemporaI graph formed by the image sequence. For swmenting long image sequences,' we have developed a recursire update procedure that incorporates knowledge of segmentation in previous frames for efficiently finding the group correspondence in the new frame.

### Citations

8073 | Maximum Likelihood from Incomplete Data via the EM algorithm
- A, Rubin
- 1977
(Show Context)
Citation Context ... (so-called "layers "). This idea has evolved through a number of papers [3, 22, 6, 20, 11, 12]. Perhaps the cleanest current formulations are based on using the ExpectationMaximization (EM)=-= algorithm[7]-=-. Typically the motion models are 2D parametric models, translational, affine or projective, the E-step is used to solve for the layers given the motions, and the M-step for solving for the motions gi... |

2580 | Normalized Cuts and Image Segmentation
- Shi, Malik
- 2000
(Show Context)
Citation Context ...the `big picture' level, rather like a painter first marking out the major areas and then filling in the details. This idea can be formalized using a graph partitioning criterion called normalized cut=-=[16]-=-. Given a motion sequence, a weighted graph G = (V; E) is constructed by taking each pixel as a node, and connecting nodes that are in a spatiotemporal neighborhood of each other. The weight on each g... |

1963 | An iterative image registration technique with an application to stereo vision
- Lucas, Kanade
- 1981
(Show Context)
Citation Context ...sequence is represented by optical flow, which can be estimated by using the outputs of spatiotemporal filters [1, 9, 8], or by using differential techniques based the brightness constancy assumption =-=[14]-=-. Although these two techniques differ in the details of their formulation, fundamentally they are equivalent[18]. The basic limitations of those techniques are also quite similar -- one can not deter... |

1953 |
Matrix computations
- Golub, Loan
- 1996
(Show Context)
Citation Context ...<0 di = b( P xi<0 di + b P xi<0 di) = b1 T D1: with the condition y i 2f1;,bgand y T D1 =0. di =0 (4) minxNcut(x) =miny yT (D , W )y yT ; (5) Dy Note that the above expression is the Rayleigh quotient=-=[GL89]-=-. If y is relaxed to take on real values, we can minimize equation (5) by solving the generalized eigenvalue system, (D , W)y = Dy: (6) However, we havetwo constraints on y, which come from the condit... |

1730 | Determining optical flow
- Horn, Schunck
- 1981
(Show Context)
Citation Context ...strict their algorithms to be run only at the places where velocity can be computed reliably[17], while others impose a smoothness constraint and apply regularization to obtain a smooth looking output=-=[10, 2, 15, 4]-=-. Alternatively, one could combine the process of motion measurement with image segmentation as has been done successfully in recent layer based approaches to motion analysis in the EM framework. In o... |

1467 | et C. TOMASI : Good features to track
- SHI
- 1994
(Show Context)
Citation Context ...performs poorly. There have been various attempts to fix these problems in optical flow computation. Some restrict their algorithms to be run only at the places where velocity can be computed reliably=-=[17]-=-, while others impose a smoothness constraint and apply regularization to obtain a smooth looking output[10, 2, 15, 4]. Alternatively, one could combine the process of motion measurement with image se... |

594 | Spatiotemporal energy models for the perception of motion
- Adelson, Bergen
- 1985
(Show Context)
Citation Context ... of motion profile. 3 Motion Profile Traditionally, local motion information in an image sequence is represented by optical flow, which can be estimated by using the outputs of spatiotemporal filters =-=[1, 9, 8]-=-, or by using differential techniques based the brightness constancy assumption [14]. Although these two techniques differ in the details of their formulation, fundamentally they are equivalent[18]. T... |

485 |
Partitioning sparse matrices with eigenvectors of graphs
- Pothen, Horst, et al.
- 1990
(Show Context)
Citation Context ...or of equation (7) with 1 1 , , eigenvalue of 0. Furthermore, D 2 (D , W)D 2 is symmetric semi-positive de nite, since 8s(D , W), also called the Laplacian matrix, is known to be semi-positive de nite=-=[PSL90]-=-. Hence z 0 is in fact the smallest eigenvector of equation (7), and all eigenvectors of equation (7) are perpendicular to each other. In particular, z 1 the second smallest eigenvector is perpendicul... |

455 | Representing Moving Images with Layers
- Wang, Adelson
- 1994
(Show Context)
Citation Context ...framework has appeared based on the idea of simultaneous estimation of multiple global motion models and their spatial supports (so-called "layers "). This idea has evolved through a number =-=of papers [3, 22, 6, 20, 11, 12]-=-. Perhaps the cleanest current formulations are based on using the ExpectationMaximization (EM) algorithm[7]. Typically the motion models are 2D parametric models, translational, affine or projective,... |

451 | A computational framework and an algorithm for the measurement of visual motion - Anandan - 1989 |

297 |
Computation of component image velocity from local phase information
- Fleet, Jepson
- 1990
(Show Context)
Citation Context ... of motion profile. 3 Motion Profile Traditionally, local motion information in an image sequence is represented by optical flow, which can be estimated by using the outputs of spatiotemporal filters =-=[1, 9, 8]-=-, or by using differential techniques based the brightness constancy assumption [14]. Although these two techniques differ in the details of their formulation, fundamentally they are equivalent[18]. T... |

268 | An optimal graph theoretic approach to data clustering: Theory and its application to image segmentation
- Wu, Leahy
- 1993
(Show Context)
Citation Context ...es this cut value. Although there are exponential number of such partitions, nding the minimum cut of a graph is a well studied problem, and there exist e cient algorithms for solving it. Wu and Leahy=-=[WL93]-=- proposed a clustering method based on this minimum cut criterion. In particular, they seek to partition a graph into k-subgraphs, such that the maximum cut across the subgroups is minimized. This pro... |

222 |
A property of eigenvectors of nonnegative symmetric matrices and its application to graph theory
- Fiedler
- 1975
(Show Context)
Citation Context ...ut value exceeds certain limit. 2.3 Related graph partition algorithms The idea of using eigenvalue problems for nding partitions of graphs originated in the work of Donath & Ho man[DH73], and Fiedler=-=[Fie75]-=-. Fiedler suggested that the eigenvector with the second smallest eigenvalue of the system (D , W)x = x could be used to split a graph. In fact the second smallest eigenvalue is called the Fiedler val... |

215 |
An investigation of smoothness constraints for the estimation of displacement vector fields from image sequences
- Nagel, Enkelmann
- 1986
(Show Context)
Citation Context ...strict their algorithms to be run only at the places where velocity can be computed reliably[17], while others impose a smoothness constraint and apply regularization to obtain a smooth looking output=-=[10, 2, 15, 4]-=-. Alternatively, one could combine the process of motion measurement with image segmentation as has been done successfully in recent layer based approaches to motion analysis in the EM framework. In o... |

196 | Layered representation of motion video using robust maximum-likelihood estimation of mixture models and mdl encoding
- Ayer, Sawhney
- 1995
(Show Context)
Citation Context ...framework has appeared based on the idea of simultaneous estimation of multiple global motion models and their spatial supports (so-called "layers "). This idea has evolved through a number =-=of papers [3, 22, 6, 20, 11, 12]-=-. Perhaps the cleanest current formulations are based on using the ExpectationMaximization (EM) algorithm[7]. Typically the motion models are 2D parametric models, translational, affine or projective,... |

160 | Smoothness in layers: Motion segmentation using nonparametric mixture estimation
- Weiss
- 1997
(Show Context)
Citation Context ...t are extracted provide the desired scene segmentation. On the other hand, the assumption that image sequence have to follow a global rigid planar motion is clearly too restrictive, and recently Weiss=-=[21]-=- has developed a variation of EM approach that is based on a non-parametric mixture model using a probability distribution over flow fields that favors smooth flow fields. When only sparse point corre... |

155 | A unified mixture framework for motion segmentation: Incorporating spatial coherence and estimating the number of models
- Weiss, Adelson
- 1996
(Show Context)
Citation Context ...framework has appeared based on the idea of simultaneous estimation of multiple global motion models and their spatial supports (so-called "layers "). This idea has evolved through a number =-=of papers [3, 22, 6, 20, 11, 12]-=-. Perhaps the cleanest current formulations are based on using the ExpectationMaximization (EM) algorithm[7]. Typically the motion models are 2D parametric models, translational, affine or projective,... |

146 | Mixture models for optical flow computation
- Jepson, Black
- 1993
(Show Context)
Citation Context |

145 |
Optical flow using spatiotemporal filters
- Heeger
- 1988
(Show Context)
Citation Context ... of motion profile. 3 Motion Profile Traditionally, local motion information in an image sequence is represented by optical flow, which can be estimated by using the outputs of spatiotemporal filters =-=[1, 9, 8]-=-, or by using differential techniques based the brightness constancy assumption [14]. Although these two techniques differ in the details of their formulation, fundamentally they are equivalent[18]. T... |

143 |
Lower bounds for the partitioning of graphs
- Donath, Hoffman
- 1973
(Show Context)
Citation Context ...n stops once the Ncut value exceeds certain limit. 2.3 Related graph partition algorithms The idea of using eigenvalue problems for nding partitions of graphs originated in the work of Donath & Ho man=-=[DH73]-=-, and Fiedler[Fie75]. Fiedler suggested that the eigenvector with the second smallest eigenvalue of the system (D , W)x = x could be used to split a graph. In fact the second smallest eigenvalue is ca... |

107 | Robust dynamic motion estimation over time
- Black, Anandan
- 1991
(Show Context)
Citation Context ...strict their algorithms to be run only at the places where velocity can be computed reliably[17], while others impose a smoothness constraint and apply regularization to obtain a smooth looking output=-=[10, 2, 15, 4]-=-. Alternatively, one could combine the process of motion measurement with image segmentation as has been done successfully in recent layer based approaches to motion analysis in the EM framework. In o... |

99 | Laws of organization in perceptual forms - Wertheimer - 1938 |

94 |
Robust estimation of a multi-layered motion representation
- Darrell, Pentland
- 1991
(Show Context)
Citation Context |

75 | Determining optical ow - Horn, Schunck - 1981 |

67 |
Surface orientation and time to contact from image divergence and deformation
- Cipolla, Blake
- 1992
(Show Context)
Citation Context ...y more robust and could be used for estimating gross measures such as divergence, deformation, rotation which have been shown to be useful variables for visual guidance of locomotion and manipulation =-=[13, 5]-=-. Snakes have been used in the computer vision literature[5] previously for this purpose. They are computationally efficient but difficult to initialize. The paper is organized as follows. Section 2 b... |

59 |
An e cient heuristic procedure for partitioning graphs
- Kernighan, Lin
- 1970
(Show Context)
Citation Context ... are shown in gure (13). segmentation back into the full resolution image space, we can use simple local search algorithms from the graph partitioning community such as the Kernighan and Lin algorithm=-=[KL70]-=- to re ne the boundaries in our segmentation. 2. Incorporating multiple cues. Our grouping framework permits us to de ne the weight of a graph edge connecting two pixels in the image sequence using co... |

40 |
An improved spectral bisection algorithm and its application to dynamic load balancing
- Driessche, Roose
- 1995
(Show Context)
Citation Context ...d both techniques to the image segmentation problem, we found that the normalized cut produces better results in practice. The generalized eigenvalue approach was rst applied to graph partitioning by =-=[DR95]-=- for dynamically balancing computational load in a parallel computer. Their algorithm is motivated by [KYSK84]'s paper on representing a hypergraph in a Euclidean Space. 3 Motion Segmentation To apply... |

33 | Accurate computation of optical flow by using layered motion representations
- Hsu, Anandan, et al.
- 1994
(Show Context)
Citation Context |

29 |
Stochastic motion clustering
- Torr, Murray
- 1994
(Show Context)
Citation Context ...ach that is based on a non-parametric mixture model using a probability distribution over flow fields that favors smooth flow fields. When only sparse point correspondences are sought, Torr and Murray=-=[19]-=- have developed an alternative approach based on characterizing rigid motions using Fundamental matrices. In our opinion, the principal weakness of the EM approach to layered motion segmentation is in... |

17 | Mixture models for optical ow computation - Jepson, Black - 1993 |

17 |
Disk packings and planar separators
- Spielman, Teng
- 1996
(Show Context)
Citation Context ...partition of V , P =(A; V , A) is de ned as cut(A;V ,A) min(jAj;jV ,Aj) . It was shown that if the Fiedler value is small, partitioning graph based on the Fiedler vector will lead to good ratio of cut=-=[ST96]-=-. Our derivation in section 2.1 can be adapted (by replacing the matrix D in the denominators by the identity matrix I) toshow that the Fiedler vector is a real valued solution to the problem of minA ... |

16 |
of organization in perceptual forms
- Laws
- 1938
(Show Context)
Citation Context ...hat incorporates knowledge of segmentation in previous frames for efficiently finding the group correspondence in the new frame. 1 Introduction Grouping based on common motion, or what the Gestaltists=-=[23] called th-=-e factor of "Common Fate", is one of the strongest cues for segmenting an image sequence into separate objects. However, implementing this perceptual capability has proved to be very challen... |

9 | A uni ed mixture framework for motion segmentation: incorporating spatial coherence and estimating the number of models - Weiss, Adelson - 1996 |

6 | Accurate computation of optical ow by using layered motion representation - Hsu, Anandan, et al. - 1994 |

3 | Optical ow from spatiotemporal lters - Heeger - 1987 |

2 |
ªA Representation of Hypergraphs in the Euclidean Space,º
- Fukunaga, Yamada, et al.
- 1984
(Show Context)
Citation Context ...s in practice. The generalized eigenvalue approach was rst applied to graph partitioning by [DR95] for dynamically balancing computational load in a parallel computer. Their algorithm is motivated by =-=[KYSK84]-=-'s paper on representing a hypergraph in a Euclidean Space. 3 Motion Segmentation To apply the normalized cut approach to motion segmentation, we treat the image sequence as a spatiotemporal data set.... |

1 |
A computationalframework and an algorithm for the measurement of vision motion
- Anandan
- 1989
(Show Context)
Citation Context |

1 |
Distributed Representation and Analysis of Vision Motion
- Simoncelli
- 1993
(Show Context)
Citation Context ..., 9, 8], or by using differential techniques based the brightness constancy assumption [14]. Although these two techniques differ in the details of their formulation, fundamentally they are equivalent=-=[18]-=-. The basic limitations of those techniques are also quite similar -- one can not determine the image velocity reliably at locations where the intensity profile is flat, such as the image of a feature... |

1 |
A Sourcebook of Gestalt Psycychology(Partial translation), chapter Laws of Organization
- Wertheimer
- 1938
(Show Context)
Citation Context ...Keywords: motion segmentation, grouping, graph partition, tracking Supported by (ARO) DAAH04-96-1-0341 y NSF Graduate Fellowships1 Introduction Grouping based on common motion, or what the Gestaltists=-=[Wer38]-=- called the factor of \Common Fate", is one of the strongest cues for segmenting an image sequence into separate objects. However, implementing this perceptual capability has proved to be very challen... |

1 | patio:emporal energy models for the percop/sion of motion - Addson, Bergen - 1985 |

1 | A computationalfi'ameworkand an algorithm for the measm'ement of vision motion - Auandan - 1989 |

1 | Layered represen_ tatlon of motion video using robust maximum4ikellhood estimation of mbcture models and MDL encoding - Sawhney - 1995 |

1 | D.B.B.nbin: Maximum likelihood t]com incomplete data via the ]EM algorithm - Dompeter, Laird - 1977 |

1 | Optical flow using spatiotemporalfilters - Hoeget - 1990 |

1 | Opticai flow. Vision fesearch - Koenderink - 1986 |

1 | An ira?erisa*ion of sincol, hhess constraints for the estimation of displacement vector fields from image sequences - Enkelmann - 1986 |

1 | Normalized cuts and image segments tion - Shi, Malik - 1997 |

1 | Distributed J,epresentation and Analysis of Vision Motion - Simoncelli - 1993 |

1 | A unified mixture framework for motion segmentation: Incorporating spatial coherence and estlmatlngthe number of models - Addson - 1996 |