## For Most Large Underdetermined Systems of Linear Equations the Minimal ℓ1-norm Solution is also the Sparsest Solution (2004)

Venue: | Comm. Pure Appl. Math |

Citations: | 343 - 9 self |

### BibTeX

@ARTICLE{Donoho04formost,

author = {David L. Donoho},

title = {For Most Large Underdetermined Systems of Linear Equations the Minimal ℓ1-norm Solution is also the Sparsest Solution},

journal = {Comm. Pure Appl. Math},

year = {2004},

volume = {59},

pages = {797--829}

}

### Years of Citing Articles

### OpenURL

### Abstract

We consider linear equations y = Φα where y is a given vector in R n, Φ is a given n by m matrix with n < m ≤ An, and we wish to solve for α ∈ R m. We suppose that the columns of Φ are normalized to unit ℓ 2 norm 1 and we place uniform measure on such Φ. We prove the existence of ρ = ρ(A) so that for large n, and for all Φ’s except a negligible fraction, the following property holds: For every y having a representation y = Φα0 by a coefficient vector α0 ∈ R m with fewer than ρ · n nonzeros, the solution α1 of the ℓ 1 minimization problem min �x�1 subject to Φα = y is unique and equal to α0. In contrast, heuristic attempts to sparsely solve such systems – greedy algorithms and thresholding – perform poorly in this challenging setting. The techniques include the use of random proportional embeddings and almost-spherical sections in Banach space theory, and deviation bounds for the eigenvalues of random Wishart matrices.

### Citations

1652 | Atomic decomposition by basis pursuit
- Chen, Donoho, et al.
- 2001
(Show Context)
Citation Context ...al orthonormal bases by Coifman and collaborators [4, 5] and combinations of several frames in by Mallat and Zhang’s work on Matching Pursuit [19], and by Chen, Donoho, and Saunders in the mid 1990’s =-=[3]-=-. A theoretical perspective showing that there is a sound mathematical basis for overcomplete representation has come together rapidly in recent years, see [7, 8, 12, 14, 16, 26, 27]. An early 2sresul... |

1298 | Robust uncertainty principles: exact signal reconstruction from highly incomplete frequency information
- Candes, Romberg, et al.
- 2006
(Show Context)
Citation Context ...ut n/5 nonzeros in a 2-fold overcomplete representation. Hence, empirically, even a mildly sparse representation could be exactly recovered by ℓ 1 optimization. Very recently, Candès, Romberg and Tao =-=[2]-=- showed that for partial Fourier systems, formed by taking n rows at random from an m-by-m standard Fourier matrix, the resulting n by m matrix with overwhelming probability allowed exact equivalence ... |

1048 | Matching pursuits with time-frequency dictionaries
- Mallat, Zhang
- 1993
(Show Context)
Citation Context ...ained using in the early 1990’s, eg. combinations of several orthonormal bases by Coifman and collaborators [4, 5] and combinations of several frames in by Mallat and Zhang’s work on Matching Pursuit =-=[19]-=-, and by Chen, Donoho, and Saunders in the mid 1990’s [3]. A theoretical perspective showing that there is a sound mathematical basis for overcomplete representation has come together rapidly in recen... |

525 | Greed is good: Algorithmic results for sparse approximation
- Tropp
- 2004
(Show Context)
Citation Context ...asurements, store less data, or investigate fewer genes. The search for sparse solutions can transform the problem completely, in many cases making unique solution possible (Lemma 2.1 below, see also =-=[7, 8, 16, 14, 26, 27]-=-). Unfortunately, this only seems to change the problem from an impossible one to an intractable one! Finding the sparsest solution to an general underdetermined system of equations is NP-hard [21]; m... |

326 |
The Concentration of Measure Phenomenon
- Ledoux
- 2001
(Show Context)
Citation Context ...bles, Lipschitz with respect to the standard Euclidean metric, with Lipschitz constant 1/ √ n. Moreover maxi Ri itself is such a Lipschitz function. By concentration of measure for Gaussian variables =-=[18]-=-, (3.3) follows. The proof of Lemma 3.4 depends on the observation – see Szarek [25], Davidson-Szarek [6] or El Karoui [13] – that the singular values of Gaussian matrices obey concentration of measur... |

317 |
Sparse approximate solutions to linear systems
- Natarajan
- 1995
(Show Context)
Citation Context ...26, 27]). Unfortunately, this only seems to change the problem from an impossible one to an intractable one! Finding the sparsest solution to an general underdetermined system of equations is NP-hard =-=[21]-=-; many classic combinatorial optimization problems can be cast in that form. In this paper we will see that for ‘most’ underdetermined systems of equations, when a sufficiently sparse solution exists,... |

283 |
Eigenvalues and Condition Numbers of Random Matrices
- Edelman
- 1988
(Show Context)
Citation Context ... GI = Φ T I ΦI with smallest eigenvalue. This eigenvector will be a random uniform point on S k−1 and so �δI�1 = � 2 � |I|�δI�2(1 + op(1)). π 24sIt generates vI = ΦIδI with Letting ρ = |I|/n, we have =-=[11]-=- �vI�2 = λ 1/2 min �δI�2. λmin = (1 − ρ 1/2 ) 2 · (1 + op(1)). Now vI is a random point on S n−1 , independent of φi for i ∈ I c . Considering the program min �δIc�1 subject to ΦIcδI c = −vI we see th... |

282 |
The Volume of Convex Bodies and Banach Space Geometry. Cambridge Tracts
- Pisier
- 1989
(Show Context)
Citation Context ... we have for n > n0 that which implies an n1 so that QED 4 Almost-Spherical Sections log(P (Ω c n,m,ρ1,λ )) ≤ AnH(ρ1/A)(1 + o(1)) − βn, P (Ω c n,m,ρ,λ ) ≤ exp(−β1n), n > n1(ρ, λ). Dvoretsky’s theorem =-=[10, 22]-=- says that every infinite-dimensional Banach space contains very high-dimensional subspaces on which the Banach norm is nearly proportional to the Euclidean norm. This is called the spherical sections... |

214 |
Sparse representations in unions of bases
- Gribonval, Nielsen
(Show Context)
Citation Context ...asurements, store less data, or investigate fewer genes. The search for sparse solutions can transform the problem completely, in many cases making unique solution possible (Lemma 2.1 below, see also =-=[7, 8, 16, 14, 26, 27]-=-). Unfortunately, this only seems to change the problem from an impossible one to an intractable one! Finding the sparsest solution to an general underdetermined system of equations is NP-hard [21]; m... |

157 | On sparse representations in arbitrary redundant bases
- Fuchs
(Show Context)
Citation Context ...asurements, store less data, or investigate fewer genes. The search for sparse solutions can transform the problem completely, in many cases making unique solution possible (Lemma 2.1 below, see also =-=[7, 8, 16, 14, 26, 27]-=-). Unfortunately, this only seems to change the problem from an impossible one to an intractable one! Finding the sparsest solution to an general underdetermined system of equations is NP-hard [21]; m... |

143 | On projection algorithms for solving convex feasibility problems
- Bauschke, Borwein
- 1996
(Show Context)
Citation Context ... to create a dual feasible point y starting from a nearby almostfeasible point y0. It is an instance of the successive projection method for finding feasible points for systems of linear inequalities =-=[1]-=-. Let I0 be the collection of indices 1 ≤ i ≤ m with and then set |〈φi, y0〉| > 1/2, y1 = y0 − PI0 y0, where PI0 denotes the least-squares projector ΦI0 (ΦT I0ΦI0 )−1ΦT . In effect, we identify the ind... |

135 |
Asymptotic theory of finite-dimensional normed spaces
- MILMAN, G
- 1986
(Show Context)
Citation Context ...I for |I| < ρn affords a spherical section of the ℓ 1 n ball. The basic argument we use derives from refinements of Dvoretsky’s theorem in Banach space theory, going back to work of Milman and others =-=[15, 24, 20]-=- Definition 4.1 Let |I| = k. We say that ΦI offers an ɛ-isometry between ℓ 2 (I) and ℓ 1 n if (1 − ɛ) · �α�2 ≤ Remarks: 1. The scale factor � π � π 2n · �ΦIα�1 ≤ (1 + ɛ) · �α�2, ∀α ∈ R k . (4.1) embed... |

116 | Empirical Processes: Theory and Applications - Pollard - 1990 |

108 |
Local operator theory, random matrices and Banach spaces
- Davidson, Szarek
- 2001
(Show Context)
Citation Context ...xi Ri itself is such a Lipschitz function. By concentration of measure for Gaussian variables [18], (3.3) follows. The proof of Lemma 3.4 depends on the observation – see Szarek [25], Davidson-Szarek =-=[6]-=- or El Karoui [13] – that the singular values of Gaussian matrices obey concentration of measure: Lemma 3.5 Let X be an n by k matrix of iid N(0, 1 n ) Gaussians, k < n. Let sℓ(X) denote the ℓ-th larg... |

103 |
Entropy Based Algorithms for Best Basis Selection
- Coifman, Wickerhauser
- 1992
(Show Context)
Citation Context ...or this viewpoint were first obtained empirically, where representations of signals were obtained using in the early 1990’s, eg. combinations of several orthonormal bases by Coifman and collaborators =-=[4, 5]-=- and combinations of several frames in by Mallat and Zhang’s work on Matching Pursuit [19], and by Chen, Donoho, and Saunders in the mid 1990’s [3]. A theoretical perspective showing that there is a s... |

90 | Just relax: Convex programming methods for subset selection and sparse approximation
- Tropp
- 2004
(Show Context)
Citation Context ...e stepwise regression; the same procedure is called Orthogonal Matching Pursuit in signal analysis, and called greedy approximation in the approximation theory literature. For further discussion, see =-=[9, 26, 27]-=-. Under sufficiently strong conditions, both methods can work. Theorem 3 (Tropp [26]) Suppose that the dictionary Φ has coherence M = maxi�=j |〈φi, φj〉|. Suppose that α0 has k ≤ M −1 /2 nonzeros, and ... |

20 | Optimally Sparse Representation from Overcomplete Dictionaries via l1-norm minimization - Donoho, Elad - 2002 |

17 |
Random embeddings of euclidean spaces in sequence spaces, Israel Journal of Mathematics 40
- Schechtman
- 1981
(Show Context)
Citation Context ...I for |I| < ρn affords a spherical section of the ℓ 1 n ball. The basic argument we use derives from refinements of Dvoretsky’s theorem in Banach space theory, going back to work of Milman and others =-=[15, 24, 20]-=- Definition 4.1 Let |I| = k. We say that ΦI offers an ɛ-isometry between ℓ 2 (I) and ℓ 1 n if (1 − ɛ) · �α�2 ≤ Remarks: 1. The scale factor � π � π 2n · �ΦIα�1 ≤ (1 + ɛ) · �α�2, ∀α ∈ R k . (4.1) embed... |

9 |
Spaces with large distance to ℓ n ∞ and random matrices
- Szarek
- 1990
(Show Context)
Citation Context ...nt 1/ √ n. Moreover maxi Ri itself is such a Lipschitz function. By concentration of measure for Gaussian variables [18], (3.3) follows. The proof of Lemma 3.4 depends on the observation – see Szarek =-=[25]-=-, Davidson-Szarek [6] or El Karoui [13] – that the singular values of Gaussian matrices obey concentration of measure: Lemma 3.5 Let X be an n by k matrix of iid N(0, 1 n ) Gaussians, k < n. Let sℓ(X)... |

6 |
Bruckstein A M 2002 A generalized uncertainty principle and sparse representation in pairs
- Elad
(Show Context)
Citation Context ..., Donoho, and Saunders in the mid 1990’s [3]. A theoretical perspective showing that there is a sound mathematical basis for overcomplete representation has come together rapidly in recent years, see =-=[7, 8, 12, 14, 16, 26, 27]-=-. An early 2sresult was the following: suppose that Φ is the concatenation of two orthobases, so that m = 2n. Suppose that the coherence - the maximal inner product between any pair of columns of Φ - ... |

5 |
Xiaoming (2001) Uncertainty Principles and Ideal Atomic Decomposition
- Donoho, Huo
- 2001
(Show Context)
Citation Context |

5 |
Some results on convex bodies and Banach spaces
- Dvoretsky
- 1961
(Show Context)
Citation Context ... we have for n > n0 that which implies an n1 so that QED 4 Almost-Spherical Sections log(P (Ω c n,m,ρ1,λ )) ≤ AnH(ρ1/A)(1 + o(1)) − βn, P (Ω c n,m,ρ,λ ) ≤ exp(−β1n), n > n1(ρ, λ). Dvoretsky’s theorem =-=[10, 22]-=- says that every infinite-dimensional Banach space contains very high-dimensional subspaces on which the Banach norm is nearly proportional to the Euclidean norm. This is called the spherical sections... |

3 |
The dimension of almost-spherical sections of convex bodies
- Figiel, Lindenstrauss, et al.
- 1977
(Show Context)
Citation Context ...I for |I| < ρn affords a spherical section of the ℓ 1 n ball. The basic argument we use derives from refinements of Dvoretsky’s theorem in Banach space theory, going back to work of Milman and others =-=[15, 24, 20]-=- Definition 4.1 Let |I| = k. We say that ΦI offers an ɛ-isometry between ℓ 2 (I) and ℓ 1 n if (1 − ɛ) · �α�2 ≤ Remarks: 1. The scale factor � π � π 2n · �ΦIα�1 ≤ (1 + ɛ) · �α�2, ∀α ∈ R k . (4.1) embed... |

2 |
Karoui (2004) New Results About Random Covariance Matrices and Statistical Applications
- El
(Show Context)
Citation Context ...uch a Lipschitz function. By concentration of measure for Gaussian variables [18], (3.3) follows. The proof of Lemma 3.4 depends on the observation – see Szarek [25], Davidson-Szarek [6] or El Karoui =-=[13]-=- – that the singular values of Gaussian matrices obey concentration of measure: Lemma 3.5 Let X be an n by k matrix of iid N(0, 1 n ) Gaussians, k < n. Let sℓ(X) denote the ℓ-th largest singular value... |

1 |
Embedding ℓm p into ℓn 1
- Johnson, Schechtman
- 1982
(Show Context)
Citation Context ...neous Isometry Our approach is based on a result for individual I, which will later be extended to get a result for every I. This individual result is well known in Banach space theory, going back to =-=[24, 17, 15]-=-. For our proof, we repackage key elements from the proof of Theorem 4.4 in Pisier’s book [22]. Pisier’s argument shows that for one specific I, there is a positive probability that ΦI offers an ɛ-iso... |