Results 1  10
of
46
LargeScale Manifold Learning
"... This paper examines the problem of extracting lowdimensional manifold structure given millions of highdimensional face images. Specifically, we address the computational challenges of nonlinear dimensionality reduction via Isomap and Laplacian Eigenmaps, using a graph containing about 18 million nod ..."
Abstract

Cited by 51 (7 self)
 Add to MetaCart
(Show Context)
This paper examines the problem of extracting lowdimensional manifold structure given millions of highdimensional face images. Specifically, we address the computational challenges of nonlinear dimensionality reduction via Isomap and Laplacian Eigenmaps, using a graph containing about 18 million nodes and 65 million edges. Since most manifold learning techniques rely on spectral decomposition, we first analyze two approximate spectral decomposition techniques for large dense matrices (Nyström and Columnsampling), providing the first direct theoretical and empirical comparison between these techniques. We next show extensive experiments on learning lowdimensional embeddings for two large face datasets: CMUPIE (35 thousand faces) and a web dataset (18 million faces). Our comparisons show that the Nyström approximation is superior to the Columnsampling method. Furthermore, approximate Isomap tends to perform better than Laplacian Eigenmaps on both clustering and classification with the labeled CMUPIE dataset. 1.
Vector diffusion maps and the connection laplacian
 CComm. Pure Appl. Math
"... Abstract. We introduce vector diffusion maps (VDM), a new mathematical framework for organizing and analyzing massive high dimensional data sets, images and shapes. VDM is a mathematical and algorithmic generalization of diffusion maps and other nonlinear dimensionality reduction methods, such as L ..."
Abstract

Cited by 48 (13 self)
 Add to MetaCart
Abstract. We introduce vector diffusion maps (VDM), a new mathematical framework for organizing and analyzing massive high dimensional data sets, images and shapes. VDM is a mathematical and algorithmic generalization of diffusion maps and other nonlinear dimensionality reduction methods, such as LLE, ISOMAP and Laplacian eigenmaps. While existing methods are either directly or indirectly related to the heat kernel for functions over the data, VDM is based on the heat kernel for vector fields. VDM provides tools for organizing complex data sets, embedding them in a low dimensional space, and interpolating and regressing vector fields over the data. In particular, it equips the data with a metric, which we refer to as the vector diffusion distance. In the manifold learning setup, where the data set is distributed on (or near) a low dimensional manifold M d embedded in R p, we prove the relation between VDM and the connectionLaplacian operator for vector fields over the manifold. Key words. Dimensionality reduction, vector fields, heat kernel, parallel transport, local principal component analysis, alignment. 1. Introduction. Apopularwaytodescribethe
Constructing Laplace Operator from Point Clouds in R^d
, 2009
"... We present an algorithm for approximating the LaplaceBeltrami operator from an arbitrary point cloud obtained from a kdimensional manifold embedded in the ddimensional space. We show that this PCD Laplace (PointCloud Data Laplace) operator converges to the LaplaceBeltrami operator on the underl ..."
Abstract

Cited by 39 (5 self)
 Add to MetaCart
(Show Context)
We present an algorithm for approximating the LaplaceBeltrami operator from an arbitrary point cloud obtained from a kdimensional manifold embedded in the ddimensional space. We show that this PCD Laplace (PointCloud Data Laplace) operator converges to the LaplaceBeltrami operator on the underlying manifold as the point cloud becomes denser. Unlike the previous work, we do not assume that the data samples are independent identically distributed from a probability distribution and do not require a global mesh. The resulting algorithm is easy to implement. We present experimental results indicating that even for point sets sampled from a uniform distribution, PCD Laplace converges faster than the weighted graph Laplacian. We also show that using our PCD Laplacian we can directly estimate certain geometric invariants, such as manifold area.
A boosting framework for visualitypreserving distance metric learning and its application to medical image retrieval
 IEEE TPAMI
, 2010
"... Abstract—Similarity measurement is a critical component in contentbased image retrieval systems, and learning a good distance metric can significantly improve retrieval performance. However, despite extensive study, there are several major shortcomings with the existing approaches for distance metr ..."
Abstract

Cited by 30 (5 self)
 Add to MetaCart
(Show Context)
Abstract—Similarity measurement is a critical component in contentbased image retrieval systems, and learning a good distance metric can significantly improve retrieval performance. However, despite extensive study, there are several major shortcomings with the existing approaches for distance metric learning that can significantly affect their application to medical image retrieval. In particular, “similarity ” can mean very different things in image retrieval: resemblance in visual appearance (e.g., two images that look like one another) or similarity in semantic annotation (e.g., two images of tumors that look quite different yet are both malignant). Current approaches for distance metric learning typically address only one goal without consideration of the other. This is problematic for medical image retrieval where the goal is to assist doctors in decision making. In these applications, given a query image, the goal is to retrieve similar images from a reference library whose semantic annotations could provide the medical professional with greater insight into the possible interpretations of the query image. If the system were to retrieve images that did not look like the query, then users would be less likely to trust the system; on the other hand, retrieving images that appear superficially similar to the query but are semantically unrelated is undesirable because that could lead users toward an incorrect diagnosis. Hence, learning a distance metric that preserves both visual resemblance and semantic similarity is important. We emphasize that, although our study is focused on medical image retrieval, the problem addressed in this work is critical to many image retrieval systems. We present a boosting framework for distance metric learning that aims to preserve both visual and semantic similarities. The boosting framework first learns a binary representation using side information, in the form of labeled pairs, and then computes the distance as a weighted Hamming
DIFFUSION MAPS, REDUCTION COORDINATES AND LOW DIMENSIONAL REPRESENTATION OF STOCHASTIC SYSTEMS
"... The concise representation of complex high dimensional stochastic systems via a few reduced coordinates is an important problem in computational physics, chemistry and biology. In this paper we use the first few eigenfunctions of the backward FokkerPlanck diffusion operator as a coarse grained low ..."
Abstract

Cited by 21 (4 self)
 Add to MetaCart
(Show Context)
The concise representation of complex high dimensional stochastic systems via a few reduced coordinates is an important problem in computational physics, chemistry and biology. In this paper we use the first few eigenfunctions of the backward FokkerPlanck diffusion operator as a coarse grained low dimensional representation for the long term evolution of a stochastic system, and show that they are optimal under a certain mean squared error criterion. We denote the mapping from physical space to these eigenfunctions as the diffusion map. While in high dimensional systems these eigenfunctions are difficult to compute numerically by conventional methods such as finite differences or finite elements, we describe a simple computational datadriven method to approximate them from a large set of simulated data. Our method is based on defining an appropriately weighted graph on the set of simulated data, and computing the first few eigenvectors and eigenvalues of the corresponding random walk matrix on this graph. Thus, our algorithm incorporates the local geometry and density at each point into a global picture that merges in a natural way data from different simulation runs. Furthermore, we describe lifting and restriction operators between the diffusion map space and the original space. These operators facilitate the description of the coarsegrained dynamics, possibly in the form of a lowdimensional effective free energy surface parameterized by the diffusion map reduction coordinates. They also enable a systematic exploration of such effective free energy surfaces through the design of additional “intelligently biased ” computational experiments. We conclude by demonstrating our method on a few examples. Key words. Diffusion maps, dimensional reduction, stochastic dynamical systems, Fokker Planck operator, metastable states, normalized graph Laplacian. AMS subject classifications. 60H10, 60J60, 62M05
Manifold regularization and semisupervised learning: some theoretical analysis
, 2008
"... Manifold regularization (Belkin et al., 2006) is a geometrically motivated framework for machine learning within which several semisupervised algorithms have been constructed. Here we try to provide some theoretical understanding of this approach. Our main result is to expose the natural structure ..."
Abstract

Cited by 19 (0 self)
 Add to MetaCart
Manifold regularization (Belkin et al., 2006) is a geometrically motivated framework for machine learning within which several semisupervised algorithms have been constructed. Here we try to provide some theoretical understanding of this approach. Our main result is to expose the natural structure of a class of problems on which manifold regularization methods are helpful. We show that for such problems, no supervised learner can learn effectively. On the other hand, a manifold based learner (that knows the manifold or “learns ” it from unlabeled examples) can learn with relatively few labeled examples. Our analysis follows a minimax style with an emphasis on finite sample results (in terms of n: the number of labeled examples). These results allow us to properly interpret manifold regularization and related spectral and geometric algorithms in terms of their potential use in semisupervised learning.
The spectrum of kernel random matrices
, 2007
"... We place ourselves in the setting of highdimensional statistical inference, where the number of variables p in a dataset of interest is of the same order of magnitude as the number of observations n. We consider the spectrum of certain kernel random matrices, in particular n × n matrices whose (i, ..."
Abstract

Cited by 14 (3 self)
 Add to MetaCart
We place ourselves in the setting of highdimensional statistical inference, where the number of variables p in a dataset of interest is of the same order of magnitude as the number of observations n. We consider the spectrum of certain kernel random matrices, in particular n × n matrices whose (i, j)th entry is f(X ′ i Xj/p) or f(‖Xi − Xj ‖ 2 /p), where p is the dimension of the data, and Xi are independent data vectors. Here f is assumed to be a locally smooth function. The study is motivated by questions arising in statistics and computer science, where these matrices are used to perform, among other things, nonlinear versions of principal component analysis. Surprisingly, we show that in highdimensions, and for the models we analyze, the problem becomes essentially linear which is at odds with heuristics sometimes used to justify the usage of these methods. The analysis also highlights certain peculiarities of models widely studied in random matrix theory and raises some questions about their relevance as tools to model highdimensional data encountered in practice. 1
An Analysis of the Convergence of Graph Laplacians
"... Existing approaches to analyzing the asymptotics of graph Laplacians typically assume a wellbehaved kernel function with smoothness assumptions. We remove the smoothness assumption and generalize the analysis of graph Laplacians to include previously unstudied graphs including kNN graphs. We also i ..."
Abstract

Cited by 14 (0 self)
 Add to MetaCart
Existing approaches to analyzing the asymptotics of graph Laplacians typically assume a wellbehaved kernel function with smoothness assumptions. We remove the smoothness assumption and generalize the analysis of graph Laplacians to include previously unstudied graphs including kNN graphs. We also introduce a kernelfree framework to analyze graph constructions with shrinking neighborhoods in general and apply it to analyze locally linear embedding (LLE). We also describe how, for a given limit operator, desirable properties such as a convergent spectrum and sparseness can be achieved by choosing the appropriate graph construction. 1.
Convergence, Stability, and Discrete Approximation of Laplace Spectra
"... Spectral methods have been widely used in a broad range of applications fields. One important object involved in such methods is the LaplaceBeltrami operator of a manifold. Indeed, a variety of work in graphics and geometric optimization uses the eigenstructures (i.e, the eigenvalues and eigenfunc ..."
Abstract

Cited by 13 (3 self)
 Add to MetaCart
Spectral methods have been widely used in a broad range of applications fields. One important object involved in such methods is the LaplaceBeltrami operator of a manifold. Indeed, a variety of work in graphics and geometric optimization uses the eigenstructures (i.e, the eigenvalues and eigenfunctions) of the Laplace operator. Applications include mesh smoothing, compression, editing, shape segmentation, matching, parameterization, and so on. While the Laplace operator is defined (mathematically) for a smooth domain, these applications often approximate a smooth manifold by a discrete mesh. The spectral structure of the manifold Laplacian is estimated from some discrete Laplace operator constructed from this mesh. In this paper, we study the important question of how well the spectrum computed from the discrete mesh approximates the true spectrum of the manifold Laplacian. We exploit a recent result on mesh Laplacian and provide the first convergence result to relate the spectrum constructed from a general mesh (approximating an mmanifold embedded in IR d) with the true spectrum. We also study how stable these eigenvalues and their discrete approximations are when the underlying manifold is perturbed, and provide explicit bounds for the Laplacian spectra of two “close” manifolds, as well as a convergence result for their discrete approximations. Finally, we present various experimental results to demonstrate that these discrete spectra are both accurate and robust in practice.
SPECTRAL CONVERGENCE OF THE CONNECTION LAPLACIAN FROM RANDOM SAMPLES
, 1306
"... ABSTRACT. Spectral methods that are based on eigenvectors and eigenvalues of discrete graph Laplacians, such as Diffusion Maps and Laplacian Eigenmaps are extremely useful for manifold learning. It was previously shown by Belkin and Niyogi [4] that the eigenvectors and eigenvalues of the graph Lapla ..."
Abstract

Cited by 9 (4 self)
 Add to MetaCart
(Show Context)
ABSTRACT. Spectral methods that are based on eigenvectors and eigenvalues of discrete graph Laplacians, such as Diffusion Maps and Laplacian Eigenmaps are extremely useful for manifold learning. It was previously shown by Belkin and Niyogi [4] that the eigenvectors and eigenvalues of the graph Laplacian converge to the eigenfunctions and eigenvalues of the LaplaceBeltrami operator of the manifold in the limit of infinitely many uniformly sampled data points. Recently, we introduced Vector Diffusion Maps and showed that the Connection Laplacian of the tangent bundle of the manifold can be approximated from random samples. In this paper, we present a unified framework for approximating other Connection Laplacians over the manifold by considering its principle bundle structure. We prove that the eigenvectors and eigenvalues of these Laplacians converge in the limit of infinitely many random samples. Our results for spectral convergence also hold in the case where the data points are sampled from a nonuniform distribution, and for manifolds with and without boundary. 1.