## Greed is good: Algorithmic results for sparse approximation (2004)

### Cached

### Download Links

- [users.cms.caltech.edu]
- [authors.library.caltech.edu]
- [users.cms.caltech.edu]
- [www.math.princeton.edu]
- DBLP

### Other Repositories/Bibliography

Venue: | IEEE Trans. Inform. Theory |

Citations: | 524 - 6 self |

### BibTeX

@ARTICLE{Tropp04greedis,

author = {Joel A. Tropp},

title = {Greed is good: Algorithmic results for sparse approximation},

journal = {IEEE Trans. Inform. Theory},

year = {2004},

volume = {50},

pages = {2231--2242}

}

### Years of Citing Articles

### OpenURL

### Abstract

Abstract. This article presents new results on using a greedy algorithm, Orthogonal Matching Pursuit (OMP), to solve the sparse approximation problem over redundant dictionaries. It contains a single sufficient condition under which both OMP and Donoho’s Basis Pursuit paradigm (BP) can recover an exactly sparse signal. It leverages this theory to show that both OMP and BP can recover all exactly sparse signals from a wide class of dictionaries. These quasi-incoherent dictionaries offer a natural generalization of incoherent dictionaries, and the Babel function is introduced to quantify the level of incoherence. Indeed, this analysis unifies all the recent results on BP and extends them to OMP. Furthermore, the paper develops a sufficient condition under which OMP can retrieve the common atoms from all optimal representations of a nonsparse signal. From there, it argues that Orthogonal Matching Pursuit is an approximation algorithm for the sparse problem over a quasiincoherent dictionary. That is, for every input signal, OMP can calculate a sparse approximant whose error is only a small factor worse than the optimal error which can be attained with the same number of terms. 1.

### Citations

4668 |
Matrix Analysis
- Horn, Johnson
- 1985
(Show Context)
Citation Context ...ally due to Donoho and Elad [DE02]. Lemma 2.3. The squared singular values of Φm exceed (1 − µ1(m − 1)).sGREED IS GOOD 7 Proof. Consider the Gram matrix G def = (Φ∗ m Φm). The Gerˇsgorin Disc Theorem =-=[HJ85]-=- states that every eigenvalue of G lies in one of the m discs � ∆k = z : |Gkk − z| ≤ � � |Gjk| . The normalization of the atoms implies that Gkk ≡ 1. The sum is bounded above by � j�=k |Gjk| = � � � j... |

1652 | Atomic decomposition by basis pursuit
- Chen, Donoho, et al.
- 2001
(Show Context)
Citation Context ...Basis Pursuit is a more sophisticated approach, which replaces the original sparse approximation problem by a linear programming problem. Empirical evidence suggests that BP is more powerful than OMP =-=[CDS99]-=-. Meanwhile, the major advantage of Orthogonal Matching Pursuit is that it has simple, fast implementations [DMA97, GMS03]. 1.1. Major Results. I have developed theory for two distinct sparse approxim... |

1047 | Matching pursuits with time-frequency dictionaries
- Mallat, Zhang
- 1993
(Show Context)
Citation Context ...more details. The idea of using the coherence parameter to summarize a dictionary has a distinguished pedigree. Mallat and Zhang introduced it as a quantity of heuristic interest for Matching Pursuit =-=[MZ93]-=-. The first theoretical developments appeared in Donoho and Huo’s paper [DH01]. Stronger results for Basis Pursuit, phrased in terms of coherence, were provided in [EB02, DE02, GN02]. Most recently, G... |

626 | Constructive Approximation - DeVore, Lorentz - 1993 |

493 | Entropy-based algorithms for best basis selection
- Coifman, Wickerhauser
- 1992
(Show Context)
Citation Context ...ture. BOB minimizes a certain objective function over a subclass of the orthogonal bases contained in the dictionary. Then it performs the best m-term approximation with respect to the selected basis =-=[CW92]-=-. Although BOB frequently produces good results, it does not offer any guarantees on the quality of approximation. Later, Villemoes developed an algorithm for the Haar wavelet packet dictionary, that ... |

405 | Projection pursuit regression
- Friedman, Stuetzle
- 1981
(Show Context)
Citation Context ...s to zero [Jon87]. In fact, this convergence is exponential [DMA97]. Greedy techniques for sparse approximation were developed in the statistics community under the name Projection Pursuit Regression =-=[FS81]-=-. In the approximation communitity, MP is known as the Pure Greedy Algorithm [Tem02]. Qian and Chen [QC94] suggested the same algorithm for time-frequency analysis independently of Mallat and Zhang. F... |

365 | Elad.“Optimally sparse representation in general (nonorthogonal) dictionaries via 1 minimization
- Donoho, Michael
- 2003
(Show Context)
Citation Context ...fficult to compute than the coherence, it is a sharper scalpel. Donoho and Elad have defined a similar notion of generalized incoherence, but they did not develop it sufficiently for present purposes =-=[DE02]-=-. Formally, the Babel function is defined by µ1(m) def = max |Λ|=m max ψ � |〈ψ, ϕλ〉| , (2.4) where the vector ψ ranges over the atoms indexed by Ω \ Λ. The subscript in the notation serves to distingu... |

359 | Uncertainty principles and ideal atomic decomposition
- Donoho, Huo
- 2001
(Show Context)
Citation Context ...ary has a distinguished pedigree. Mallat and Zhang introduced it as a quantity of heuristic interest for Matching Pursuit [MZ93]. The first theoretical developments appeared in Donoho and Huo’s paper =-=[DH01]-=-. Stronger results for Basis Pursuit, phrased in terms of coherence, were provided in [EB02, DE02, GN02]. Most recently, Gilbert, Muthukrishnan and Strauss have exhibited an approximation algorithm fo... |

350 | Orthogonal matching pursuit: recursive function approximation with applications to wavelet decomposition - Pati, Rezaiifar, et al. - 1993 |

316 | Sparse approximate solutions to linear systems - Natarajan - 1995 |

296 | Stable recovery of sparse overcomplete representations in the presence of noise - Donoho, Elad, et al. - 2006 |

256 |
Foundations of Time-Frequency Analysis
- Gröchenig
- 2001
(Show Context)
Citation Context ...licated functions. A Gabor dictionary, for example, consists of complex exponentials at different frequencies which are localized to short time intervals. It is used for joint time-frequency analysis =-=[Grö01]-=-. Redundant systems raise the awkward question of how to use them effectively for approximation. The problem of representing a signal with the best linear combination of m elements from a dictionary i... |

222 |
Introductory Functional Analysis with Applications
- Kreyszig
- 1978
(Show Context)
Citation Context ...finition of the Babel function, � � �A�1,1 = max � k � �� ϕλk , ϕλj � ≤ µ1(m − 1). j�=k (3.7)sGREED IS GOOD 13 Whenever �A�1,1 < 1, the von Neumann series � (−A) k converges to the inverse (Im +A) −1 =-=[Kre89]-=-. In this case, we may compute � �(Φ ∗ opt Φopt) −1�� = 1,1 � �(Im + A) −1� � � ∞� � � = 1,1 � (−A) � k � � � � ≤ ≤ ∞� k=0 �A� k 1,1 = 1 1 − µ1(m − 1) . 1 k=0 1 − �A� 1,1 Introduce the estimates (3.8)... |

214 |
Sparse representations in unions of bases
- Gribonval, Nielsen
(Show Context)
Citation Context ...independent, whence Theorem 2.4 (Donoho-Elad [DE02]). Lower bounds on the spark of a dictionary are j�=k spark Φ ≥ min{m : µ1(m − 1) ≥ 1}, and spark Φ ≥ µ −1 + 1. The coherence result also appears in =-=[GN02]-=-. For structured dictionaries, better estimates are possible. For example, Theorem 2.5 (Gribonval-Nielsen [GN02]). If D is a µ-coherent dictionary consisting of L orthonormal bases, � spark Φ ≥ 1 + 1 ... |

174 | A generalized uncertainty principle and sparse representations of pairs of bases
- Elad, Bruckstein
(Show Context)
Citation Context ... Theorem 2.7 (Donoho-Huo [DH01]). Let D be a union of two orthonormal bases with coherence µ. If m < 1 2 (µ−1 + 1), then Basis Pursuit can recover any superposition of m atoms from the dictionary. In =-=[EB02]-=-, Elad and Bruckstein made some improvements to the bound on m, which turn out to be optimal [FN]. More recently, the theorem of Donoho and Huo has been extended to multiONBs and arbitrary incoherent ... |

153 |
Orthogonal Least Squares Methods and their Application to Non-Linear System Identification
- Chen, Billings, et al.
- 1989
(Show Context)
Citation Context ...k that the residual equals zero after d steps. Orthogonal Matching Pursuit was developed independently by many researchers. The earliest reference appears to be a 1989 paper of Chen, Billings and Luo =-=[CBL89]-=-. The first signal processing papers on OMP arrived around 1993 [PRK93, DMZ94]. 2.3.3. OMP and the Sparse Problem. Gilbert, Muthukrishnan and Strauss have shown that Orthogonal Matching Pursuit is an ... |

153 | Grassmannian frames with applications to coding and communication - Strohmer, Heath |

90 | Just relax: Convex programming methods for subset selection and sparse approximation - Tropp - 2004 |

79 | Block coordinate relaxation methods for nonparametric wavelet denoising
- Sardy, Brouce, et al.
- 2000
(Show Context)
Citation Context ...t problem. The original paper advocates interior-point methods of linear programming [CDS99]. More recently, Sardy, Bruce and Tseng have suggested another procedure called Block Coordinate Relaxation =-=[SBT00]-=-. Both techniques are computationally intensive. At present, the Basis Pursuit paradigm offers no approximation guarantees for the general sparse approximation problem. There is, however, a sequence o... |

75 | Adaptive time-frequency decompositions - Davis, Mallat, et al. - 1994 |

71 | Approximation of functions over redundant dictionaries using coherence
- Gilbert, Muthukrishnan, et al.
- 2003
(Show Context)
Citation Context ..., it follows that Orthogonal Matching Pursuit will calculate an m-term approximant am that satisfies �x − am�2 ≤ √ 1 + 6m �x − aopt�2 . This theorem extends work of Gilbert, Muthukrishnan and Strauss =-=[GMS03]-=-. No comparable results are available for the Basis Pursuit paradigm. 2. Background 2.1. Sparse Approximation Problems. The standard sparse approximation problem 1 is set in the Hilbert space C d . A ... |

54 |
Greedy adaptive approximation
- Davis, Mallat, et al.
- 1997
(Show Context)
Citation Context ...ctionary for many approximations. A second reason is that Davis, Mallat and Avellaneda have shown that solving or even approximating the solution of (2.1) is NP-hard if the dictionary is unrestricted =-=[DMA97]-=-. Nevertheless, it is not quixotic to seek algorithms for the sparse problem over a particular dictionary. We shall also consider a second problem called (D, m)-Exact-Sparse, where the input signal is... |

54 |
High-Dimensional Computational Geometry
- Indyk
(Show Context)
Citation Context ...ed with an approximate nearest neighbor data structure calculates m-term approximants that satisfy �x − am�2 ≤ √ 1 + 24m �x − aopt�2 . Implementing ANNOMP with Indyk’s nearest neighbor data structure =-=[Ind00]-=- requires preprocessing time and space O(N (1/η) O(d) polylog (dN)). Subsequently, each m-term representation can be calculated in O(m 2 d + md polylog (dN)) time and O(md) additional space, which is ... |

53 | On sparse representation in pairs of bases - Feuer, Nemirosky |

48 |
On a Conjecture of Huber Concerning the Convergence of Projection Pursuit Regression
- Jones
- 1987
(Show Context)
Citation Context ...tionary is an orthonormal basis, the approximant am is always an optimal m-term representation of the signal. For general dictionaries, Jones has shown that the norm of the residual converges to zero =-=[Jon87]-=-. In fact, this convergence is exponential [DMA97]. Greedy techniques for sparse approximation were developed in the statistics community under the name Projection Pursuit Regression [FS81]. In the ap... |

48 | Nonlinear methods of approximation
- Temlyakov
- 2002
(Show Context)
Citation Context ...ues for sparse approximation were developed in the statistics community under the name Projection Pursuit Regression [FS81]. In the approximation communitity, MP is known as the Pure Greedy Algorithm =-=[Tem02]-=-. Qian and Chen [QC94] suggested the same algorithm for time-frequency analysis independently of Mallat and Zhang. For more history, theory and an comprehensive list of references, see Temlyakov’s mon... |

30 |
Signal representation using adaptive normalized Gaussian functions, Signal Process
- Qian, Chen
- 1994
(Show Context)
Citation Context ...ation were developed in the statistics community under the name Projection Pursuit Regression [FS81]. In the approximation communitity, MP is known as the Pure Greedy Algorithm [Tem02]. Qian and Chen =-=[QC94]-=- suggested the same algorithm for time-frequency analysis independently of Mallat and Zhang. For more history, theory and an comprehensive list of references, see Temlyakov’s monograph [Tem02]. (2.6)s... |

23 |
Improved sparse approximation over quasiincoherent dictionaries
- Gilbert, Muthukrishnan, et al.
- 2003
(Show Context)
Citation Context ...which is quite good considering that we have placed no restrictions on the dictionary beyond quasi-incoherence. A more sophisticated greedy algorithm based on approximate nearest neighbors appears in =-=[GMST03]-=-, but no additional approximation guarantees are presently available.sGREED IS GOOD 19 Appendix A. Proof of Theorem 3.7 Theorem 3.7. Suppose that D consists of J concatenated orthonormal bases with ov... |

20 | On quasi-orthogonal signatures for CDMA systems
- Heath, Strohmer, et al.
- 2006
(Show Context)
Citation Context ... build a multi-ONB which contains d or 2 The ℓ0 quasi-norm of a vector equals the number of nonzero components.sGREED IS GOOD 5 even (d + 1) bases yet retains the minimal coherence µ = d−1/2 possible =-=[HSP02]-=-. dictionaries, a lower bound on the coherence is � For general µ ≥ N − d d (N − 1) . If each atomic inner product meets this bound, the dictionary is called an optimal Grassmannian frame. See [SH02, ... |

16 | Maximal sparsity representation via l minimization - Donoho, Elad - 2003 |

5 |
On sparse representations in pairs of bases. Accepted to
- Feuer, Nemirovsky
- 2002
(Show Context)
Citation Context ...< 1 2 (µ−1 + 1), then Basis Pursuit can recover any superposition of m atoms from the dictionary. In [EB02], Elad and Bruckstein made some improvements to the bound on m, which turn out to be optimal =-=[FN]-=-. More recently, the theorem of Donoho and Huo has been extended to multiONBs and arbitrary incoherent dictionaries [DE02, GN02]. Donoho and Elad have also developed a generalized notion of incoherenc... |

4 |
approximation with walsh atoms
- Best
- 1997
(Show Context)
Citation Context ...t offer any guarantees on the quality of approximation. Later, Villemoes developed an algorithm for the Haar wavelet packet dictionary, that produces provably good approximations with a low time cost =-=[Vil97]-=-. Villemoes’ result is a serious coup, even though Haar wavelets have limited applicability.sGREED IS GOOD 9 2.4.2. Basis Pursuit. The other major approach to sparse approximation is the Basis Pursuit... |

2 | Villemoes, “Best approximation with Walsh atoms - F - 1997 |

1 | Existence of real Grassmannian frames - Sustik, Tropp - 2003 |

1 | Nonlinear approximation,” Acta Num - DeVore - 1998 |

1 | On the Exponential Convergence of Matching Pursuits - Gribonval, Nielsen |

1 | On the stability of Basis Pursuit in the presence of noise,” working draft - Donoho, Elad |