## Random projections of smooth manifolds (2006)

### Cached

### Download Links

Venue: | Foundations of Computational Mathematics |

Citations: | 80 - 22 self |

### BibTeX

@INPROCEEDINGS{Baraniuk06randomprojections,

author = {Richard G. Baraniuk and Michael B. Wakin},

title = {Random projections of smooth manifolds},

booktitle = {Foundations of Computational Mathematics},

year = {2006},

pages = {941--944}

}

### Years of Citing Articles

### OpenURL

### Abstract

We propose a new approach for nonadaptive dimensionality reduction of manifold-modeled data, demonstrating that a small number of random linear projections can preserve key information about a manifold-modeled signal. We center our analysis on the effect of a random linear projection operator Φ: R N → R M, M < N, on a smooth well-conditioned K-dimensional submanifold M ⊂ R N. As our main theoretical contribution, we establish a sufficient number M of random projections to guarantee that, with high probability, all pairwise Euclidean and geodesic distances between points on M are well-preserved under the mapping Φ. Our results bear strong resemblance to the emerging theory of Compressed Sensing (CS), in which sparse signals can be recovered from small numbers of random linear measurements. As in CS, the random measurements we propose can be used to recover the original data in R N. Moreover, like the fundamental bound in CS, our requisite M is linear in the “information level” K and logarithmic in the ambient dimension N; we also identify a logarithmic dependence on the volume and conditioning of the manifold. In addition to recovering faithful approximations to manifold-modeled signals, however, the random projections we propose can also be used to discern key properties about the manifold. We discuss connections and contrasts with existing techniques in manifold learning, a setting where dimensionality reducing mappings are typically nonlinear and constructed adaptively from a set of sampled training data.

### Citations

2982 | Eigenfaces for Recognition
- Turk, Pentland
- 1991
(Show Context)
Citation Context ...ro for i /∈ Ω). Also, the best K-dimensional subspace to approximate a class of signals in R N can be discovered using principal components analysis (PCA) (also known as the Karhunen-Loève transform) =-=[42]-=-. 1 Although we focus on the case of an orthonormal basis for simplicity, many of the basic ideas we discuss generalize to frames and arbitrary dictionaries in R N . 2s1.2.2 Sparse models Sparse signa... |

2267 |
A wavelet tour of signal processing
- Mallat
- 1998
(Show Context)
Citation Context ...forms a K-dimensional linear subspace of R N ; examples include low-frequency subspaces in the Fourier domain or scaling spaces in the wavelet domain for representing and approximating smooth signals =-=[32]-=-. The linear geometry of these signal classes leads to simple, linear algorithms for dimensionality reduction. An ℓ2-nearest “linear approximation” to a signal x ∈ R N can be computed via orthogonal p... |

1864 | Compressed sensing
- Donoho
- 2006
(Show Context)
Citation Context ...he nonlinear geometry of ΣK. Despite the apparent need for adaptive, nonlinear methods for dimensionality reduction of sparse signals, a radically different technique known as Compressed Sensing (CS) =-=[12, 18]-=- has emerged that relies on a nonadaptive, linear method for dimensionality reduction. Like traditional approaches to approximation and compression, the goal of CS is to maintain a low-dimensional rep... |

1774 |
A global geometric framework for nonlinear dimensionality reduction, Science 290
- Tenenbaum, Silva, et al.
- 2000
(Show Context)
Citation Context ... mappings from R N to R M for some M < N (ideally M = K) that are adapted to the training data and intended to preserve some characteristic property of the manifold. Example algorithms include ISOMAP =-=[40]-=-, Hessian Eigenmaps (HLLE) [24], and Maximum Variance Unfolding (MVU) [46], which attempt to learn isometric embeddings of the manifold (preserving pairwise geodesic distances); Locally Linear Embeddi... |

1708 | Nonlinear dimensionality reduction by locally linear embedding
- Saul, Roweis
- 2000
(Show Context)
Citation Context ... Eigenmaps (HLLE) [24], and Maximum Variance Unfolding (MVU) [46], which attempt to learn isometric embeddings of the manifold (preserving pairwise geodesic distances); Locally Linear Embedding (LLE) =-=[36]-=-, which attempts to preserve local linear neighborhood structures among the embedded points; Local Tangent Space Alignment (LTSA) [47], which attempts to preserve local coordinates in each tangent spa... |

1401 | Robust uncertainty principles: Exact signal reconstruction from highly incomplete frequency information
- Candès, Romberg, et al.
- 2006
(Show Context)
Citation Context ...n the basis Ψ to identify the correct signal x from an uncountable number of possibilities. CS has many promising applications in signal acquisition, compression, medical imaging, and sensor networks =-=[3, 8, 10, 11, 22, 27, 38]-=-; the random nature of the operator Φ makes it a particularly intriguing universal measurement scheme for settings in which the basis Ψ is unknown at the encoder or multi-signal settings in which dist... |

887 | Near-optimal signal recovery from random projections: universal encoding strategies
- Candes, Tao
- 2006
(Show Context)
Citation Context ...he nonlinear geometry of ΣK. Despite the apparent need for adaptive, nonlinear methods for dimensionality reduction of sparse signals, a radically different technique known as Compressed Sensing (CS) =-=[12, 18]-=- has emerged that relies on a nonadaptive, linear method for dimensionality reduction. Like traditional approaches to approximation and compression, the goal of CS is to maintain a low-dimensional rep... |

802 |
Stable signal recovery from incomplete and inaccurate measurements
- Candès, Romberg, et al.
- 2006
(Show Context)
Citation Context ...gorithms), however, requires a more “stable” embedding (where sparse signals remain well-separated in RM ). One way of characterizing this stability is known as the Restricted Isometry Property (RIP) =-=[9, 10, 12]-=-; we say Φ has RIP of order K if for every x ∈ ΣK, (1 − ǫ) � M N ≤ �Φx� 2 �x� 2 � M ≤ (1 + ǫ) N . (Observe that the RIP of order 2K ensures that distinct K-sparse signals remain well-separated in R M ... |

781 | Laplacian eigenmaps for dimensionality reduction and data representation, Neural Comput
- Belkin, Niyogi
- 2003
(Show Context)
Citation Context ...ifolds from sampled data without constructing an explicit embedding in R M [13, 15, 33] and for constructing functions on the point samples in R N that reflect the intrinsic structure of the manifold =-=[4, 14]-=-. Naturally, the burden of storing the sampled data points and implementing any of these manifold learning algorithms increases with the native dimension N of the data. 1.3 Contributions: Universal di... |

445 | The Dantzig selector: statistical estimation when p is much larger than n,” Annals of Statistics
- Candès, Tao
(Show Context)
Citation Context ...n the basis Ψ to identify the correct signal x from an uncountable number of possibilities. CS has many promising applications in signal acquisition, compression, medical imaging, and sensor networks =-=[3, 8, 10, 11, 22, 27, 38]-=-; the random nature of the operator Φ makes it a particularly intriguing universal measurement scheme for settings in which the basis Ψ is unknown at the encoder or multi-signal settings in which dist... |

362 | 2004) For Most Large Underdetermined Systems of Linear Equations the Minimal l1 norm Solution is also the sparsest Solution. URL : http://stat.stanford.edu/˜donoho/Reports/2004 - Donoho |

346 |
JPEG2000: Image compression fundamentals, standards and practice
- Taubman, Marcellin
- 2001
(Show Context)
Citation Context ... the K largest coefficients and then setting αi to zero for i /∈ Ω). Transform coding algorithms (which form the heart of many modern signal and image compression standards such as JPEG and JPEG-2000 =-=[39]-=-) also build upon this basic idea. Again, the geometry of the sparse signal class plays an important role in the signal processing. Unlike the linear case described above, no single K-dimensional subs... |

320 | A simple proof of the restricted isometry property for random matrices,” Constructive Approximation
- Baraniuk, Davenport, et al.
(Show Context)
Citation Context ... Various other geometric arguments surrounding CS have also been made involving n-widths of ℓ p balls and the properties of randomly projected polytopes [11, 12, 17–21, 25, 34, 37]. In a recent paper =-=[2]-=-, we have identified a fundamental connection between CS and the JohnsonLindenstrauss (JL) lemma [1, 16], which concerns the stable embedding of a finite point cloud under a random dimensionality-redu... |

194 |
eigenmaps: locally linear embedding techniques for high-dimensional data
- Donoho, Grimes, et al.
- 2003
(Show Context)
Citation Context ...ome M < N (ideally M = K) that are adapted to the training data and intended to preserve some characteristic property of the manifold. Example algorithms include ISOMAP [40], Hessian Eigenmaps (HLLE) =-=[24]-=-, and Maximum Variance Unfolding (MVU) [46], which attempt to learn isometric embeddings of the manifold (preserving pairwise geodesic distances); Locally Linear Embedding (LLE) [36], which attempts t... |

177 | Signal reconstruction from noisy random projections
- Haupt, Nowak
- 2006
(Show Context)
Citation Context ...n the basis Ψ to identify the correct signal x from an uncountable number of possibilities. CS has many promising applications in signal acquisition, compression, medical imaging, and sensor networks =-=[3, 8, 10, 11, 22, 27, 38]-=-; the random nature of the operator Φ makes it a particularly intriguing universal measurement scheme for settings in which the basis Ψ is unknown at the encoder or multi-signal settings in which dist... |

174 | Unsupervised learning of image manifolds by semidefinite programming
- Weinberger, Saul
- 2006
(Show Context)
Citation Context ...to the training data and intended to preserve some characteristic property of the manifold. Example algorithms include ISOMAP [40], Hessian Eigenmaps (HLLE) [24], and Maximum Variance Unfolding (MVU) =-=[46]-=-, which attempt to learn isometric embeddings of the manifold (preserving pairwise geodesic distances); Locally Linear Embedding (LLE) [36], which attempts to preserve local linear neighborhood struct... |

170 | Charting a manifold, in
- Brand
- 2002
(Show Context)
Citation Context ... neighborhood structures among the embedded points; Local Tangent Space Alignment (LTSA) [47], which attempts to preserve local coordinates in each tangent space; and a method for charting a manifold =-=[5]-=- that attempts to preserve local neighborhood structures. These algorithms can be useful for learning the dimension and parametrizations of manifolds, for sorting data, for visualization and navigatio... |

167 | Database-friendly random projections
- Achlioptas
- 2001
(Show Context)
Citation Context ...s and the properties of randomly projected polytopes [11, 12, 17–21, 25, 34, 37]. In a recent paper [2], we have identified a fundamental connection between CS and the JohnsonLindenstrauss (JL) lemma =-=[1, 16]-=-, which concerns the stable embedding of a finite point cloud under a random dimensionality-reducing projection. Lemma 1.1 [Johnson-Lindenstrauss] Let Q be a finite collection of points in R N . Fix 0... |

155 | Modeling the Manifolds of Images of Handwritten Digits
- Hinton, Dayan, et al.
- 1997
(Show Context)
Citation Context ...n Section 1.2.2 that we also denote by K.) Low-dimensional manifolds have also been proposed as approximate models for nonparametric signal classes such as images of human faces or handwritten digits =-=[7, 28, 42]-=-. Most algorithms for dimensionality reduction of manifold-modeled signals involve “learning” the manifold structure from a collection of data points, typically by constructing nonlinear mappings from... |

147 | Principal manifolds and nonlinear dimensionality reduction by local tangent space alignment
- Zhang, Zha
- 2004
(Show Context)
Citation Context ...ving pairwise geodesic distances); Locally Linear Embedding (LLE) [36], which attempts to preserve local linear neighborhood structures among the embedded points; Local Tangent Space Alignment (LTSA) =-=[47]-=-, which attempts to preserve local coordinates in each tangent space; and a method for charting a manifold [5] that attempts to preserve local neighborhood structures. These algorithms can be useful f... |

119 | An elementary proof of the johnson-lindenstrauss lemma
- Dasgupta, Gupta
- 1999
(Show Context)
Citation Context ...s and the properties of randomly projected polytopes [11, 12, 17–21, 25, 34, 37]. In a recent paper [2], we have identified a fundamental connection between CS and the JohnsonLindenstrauss (JL) lemma =-=[1, 16]-=-, which concerns the stable embedding of a finite point cloud under a random dimensionality-reducing projection. Lemma 1.1 [Johnson-Lindenstrauss] Let Q be a finite collection of points in R N . Fix 0... |

90 | Distributed compressed sensing
- Baron, Wakin, et al.
- 2005
(Show Context)
Citation Context |

90 | Geometric approach to error correcting codes and reconstruction of signals - Rudelson, Vershynin - 2005 |

88 | 2005) Neighborly Polytopes and the Sparse Solution of Underdetermined Systems of Linear Equations. To appear - Donoho |

80 | Error correction via linear programming - Candes, Tao - 2005 |

78 | Diffusion wavelets
- Coifman, Maggioni
- 2006
(Show Context)
Citation Context ...ifolds from sampled data without constructing an explicit embedding in R M [13, 15, 33] and for constructing functions on the point samples in R N that reflect the intrinsic structure of the manifold =-=[4, 14]-=-. Naturally, the burden of storing the sampled data points and implementing any of these manifold learning algorithms increases with the native dimension N of the data. 1.3 Contributions: Universal di... |

78 | Counting faces of randomly-projected polytopes when the projection radically lowers dimension. Manuscript arXiv:math/0607364v2 [math.MG - Donoho, Tanner - 2006 |

72 | Geodesic entropic graphs for dimension and entropy estimation in manifold learning
- Costa, Hero
- 2004
(Show Context)
Citation Context ... of dynamical systems having low-dimensional attractors. Additional algorithms have also been proposed for characterizing manifolds from sampled data without constructing an explicit embedding in R M =-=[13, 15, 33]-=- and for constructing functions on the point samples in R N that reflect the intrinsic structure of the manifold [4, 14]. Naturally, the burden of storing the sampled data points and implementing any ... |

70 |
Random filters for compressive sampling and reconstruction
- Tropp, Wakin, et al.
- 2006
(Show Context)
Citation Context ...ying Φ. However, in some cases the reduced storage requirement could justify the computational burden. Moreover, to support the developing CS theory, physical “analog-to-information” hardware devices =-=[30, 41]-=- and imaging systems [38] have been proposed that directly acquire random projections of signals into R M without first sampling the signals in R N ; this eliminates the computational burden of applyi... |

61 | Neighborliness of randomly-projected simplices in high dimensions - Donoho, Tanner - 2005 |

57 | Persistence barcodes for shapes
- CARLSSON, ZOMORODIAN, et al.
(Show Context)
Citation Context ... of dynamical systems having low-dimensional attractors. Additional algorithms have also been proposed for characterizing manifolds from sampled data without constructing an explicit embedding in R M =-=[13, 15, 33]-=- and for constructing functions on the point samples in R N that reflect the intrinsic structure of the manifold [4, 14]. Naturally, the burden of storing the sampled data points and implementing any ... |

51 | High-dimensional centrally symmetric polytopes with neighborliness proportional to dimension - Donoho - 2006 |

51 | Constructive approximation, advanced problems
- Lorentz, Golitschek, et al.
- 1996
(Show Context)
Citation Context ...ts Q1(a) ⊂ Tana such that �q� 2 ≤ 1 for all q ∈ Q1(a) and such that for every u ∈ Tana with �u� 2 ≤ 1, min q∈Q1(a) �u − q�2 ≤ δ. This can be accomplished with #Q1(a) ≤ (3/δ) K (see e.g. Chapter 13 of =-=[31]-=-). We then define the renormalized set Q2(a) = {Tq : q ∈ Q1(a)} and note that �q� 2 ≤ T for all q ∈ Q2(a) and that for every u ∈ Tana with �u� 2 ≤ T, We now define the set min q∈Q2(a) �u − q�2 ≤ Tδ. (... |

46 | Multiscale representations for manifold-valued data, Multiscale Mod
- Rahman, Drori, et al.
- 2005
(Show Context)
Citation Context ...anifold-modeled data, demonstrating that small numbers of random linear projections can 3 In general, Θ itself can be a K-dimensional manifold and need not be a subset of R K . We refer the reader to =-=[35]-=- for an excellent overview of several manifolds with relevance to signal processing, including the rotation group SO(3), which can be used for representing orientations of objects in 3-D space. 5spres... |

42 | The multiscale structure of non-differentiable image manifolds
- Wakin, Donoho, et al.
- 2005
(Show Context)
Citation Context ...on such as the position of a camera or microphone recording a scene or the relative placement of objects/speakers in a scene; and • parameters governing the output of some articulated physical system =-=[23, 26, 45]-=-. In these and many other cases, the geometry of the signal class forms a nonlinear K-dimensional submanifold of R N , F = {xθ : θ ∈ Θ}, where Θ is the K-dimensional parameter space. 3 (Note the dimen... |

38 | Analog-to-information conversion via random demodulation
- Kirolos, Laska, et al.
- 2006
(Show Context)
Citation Context ...ying Φ. However, in some cases the reduced storage requirement could justify the computational burden. Moreover, to support the developing CS theory, physical “analog-to-information” hardware devices =-=[30, 41]-=- and imaging systems [38] have been proposed that directly acquire random projections of signals into R M without first sampling the signals in R N ; this eliminates the computational burden of applyi... |

38 |
Nearest neighbor preserving embeddings
- Indyk, Naor
(Show Context)
Citation Context ...may no longer be nonexpanding; however with high probability the norm �Φ� 2 can be bounded by a small constant. 6. During final preparation of this manuscript, we became aware of another recent paper =-=[30]-=- concerning embeddings of low-dimensional signal sets by random projections. While the signal classes considered in that paper are slightly more general, the conditions required on distance preservati... |

36 |
Image manifolds which are isometric to euclidean space
- Donoho, Grimes
(Show Context)
Citation Context ...on such as the position of a camera or microphone recording a scene or the relative placement of objects/speakers in a scene; and • parameters governing the output of some articulated physical system =-=[23, 26, 45]-=-. In these and many other cases, the geometry of the signal class forms a nonlinear K-dimensional submanifold of R N , F = {xθ : θ ∈ Θ}, where Θ is the K-dimensional parameter space. 3 (Note the dimen... |

29 | A New Approach for Dimensionality Reduction: Theory and Algorithms
- Broomhead, Kirby
- 1996
(Show Context)
Citation Context ...eprocessing to make further analysis more tractable; common demonstrations include analysis of face images and classification of handwritten digits. A related technique, the Whitney Reduction Network =-=[6, 7]-=-, uses a training data set to adaptively construct a linear mapping from R N to R M that attempts to preserve ambient pairwise distances on the manifold; this is particularly useful for processing the... |

28 |
The Johnson-Lindenstrauss lemma meets compressed sensing
- Baraniuk, Davenport, et al.
- 2006
(Show Context)
Citation Context .... Various other geometric arguments surrounding CS have also been made involving n-widths of ℓ p balls and the properties of randomly projected polytopes [10,11, 16–20, 24, 32, 35]. In a recent paper =-=[2]-=-, we have identified a fundamental connection between CS and the JohnsonLindenstrauss (JL) lemma [1, 15], which concerns the stable embedding of a finite point cloud under a random dimensionality-redu... |

16 |
The Whitney Reduction Network: a method for computing autoassociative graphs
- Broomhead, Kirby
- 2001
(Show Context)
Citation Context ...n Section 1.2.2 that we also denote by K.) Low-dimensional manifolds have also been proposed as approximate models for nonparametric signal classes such as images of human faces or handwritten digits =-=[7, 28, 42]-=-. Most algorithms for dimensionality reduction of manifold-modeled signals involve “learning” the manifold structure from a collection of data points, typically by constructing nonlinear mappings from... |

16 |
A compressed sensing camera: New theory and an implementation using digital micromirrors
- Takhar, Bansal, et al.
- 2006
(Show Context)
Citation Context |

15 |
Differential topology, volume 33 of Graduate Texts in Mathematics
- Hirsch
- 1994
(Show Context)
Citation Context ... of measurements so that (with high probability), 4 the operator Φ embeds M into R M . This implies also that x will be uniquely identifiable from its projection y = Φx. 4 Whitney’s Embedding Theorem =-=[29]-=- actually suggests for certain K-dimensional manifolds that the number of measurements need be no larger than 2K +1 to ensure an embedding. However, as it offers no guarantee of stability, the practic... |

13 | Random projections of signal manifolds
- Wakin, Baraniuk
- 2006
(Show Context)
Citation Context ... are preserved, one would expect the challenge of solving (13) in the measurement space R M to roughly reflect the difficulty of solving (12) in the initial ambient space R N . We refer the reader to =-=[43, 44]-=- for additional discussion of recovery algorithms in more specific contexts such as image processing. These papers also include a series of simple but promising experiments that support manifold-based... |

11 |
Decoding via Linear Programming
- Candès, Tao
- 2005
(Show Context)
Citation Context |

9 | The Geometry of Low-dimensional Signal Models
- Wakin
- 2006
(Show Context)
Citation Context ... are preserved, one would expect the challenge of solving (13) in the measurement space R M to roughly reflect the difficulty of solving (12) in the initial ambient space R N . We refer the reader to =-=[43, 44]-=- for additional discussion of recovery algorithms in more specific contexts such as image processing. These papers also include a series of simple but promising experiments that support manifold-based... |

7 |
New methods in nonlinear dimensionality reduction
- Grimes
- 2003
(Show Context)
Citation Context ...on such as the position of a camera or microphone recording a scene or the relative placement of objects/speakers in a scene; and • parameters governing the output of some articulated physical system =-=[23, 26, 45]-=-. In these and many other cases, the geometry of the signal class forms a nonlinear K-dimensional submanifold of R N , F = {xθ : θ ∈ Θ}, where Θ is the K-dimensional parameter space. 3 (Note the dimen... |

7 |
Finding the homology of submanifolds with confidence from random samples
- Niyogi, Smale, et al.
- 2009
(Show Context)
Citation Context ... of dynamical systems having low-dimensional attractors. Additional algorithms have also been proposed for characterizing manifolds from sampled data without constructing an explicit embedding in R M =-=[13, 15, 33]-=- and for constructing functions on the point samples in R N that reflect the intrinsic structure of the manifold [4, 14]. Naturally, the burden of storing the sampled data points and implementing any ... |

7 | N-widths and optimal recovery - Pinkus - 1986 |

6 |
Constructive Approximation: Advanced problems, Grundlehren vol. 304
- Makovoz
- 1996
(Show Context)
Citation Context ...nts Q1(a) ⊂ Tana such that ‖q‖2 ≤ 1 for all q ∈ Q1(a) and such that for every u ∈ Tana with ‖u‖ 2 ≤ 1, min q∈Q1(a) ‖u − q‖2 ≤ δ. This can be accomplished with #Q1(a) ≤ (3/δ) K (see e.g. Chapter 13 of =-=[29]-=-). We then define the renormalized set Q2(a) = {Tq : q ∈ Q1(a)} and note that ‖q‖ 2 ≤ T for all q ∈ Q2(a) and that for every u ∈ Tana with ‖u‖ 2 ≤ T, We now define the set min q∈Q2(a) ‖u − q‖2 ≤ Tδ. (... |