## Learning to Sense Sparse Signals: Simultaneous Sensing Matrix and Sparsifying Dictionary Optimization (2008)

Citations: | 28 - 4 self |

### BibTeX

@MISC{Duarte-carvajalino08learningto,

author = {Julio Martin Duarte-carvajalino and Guillermo Sapiro and Julio Martin Duarte-carvajalino and Guillermo Sapiro},

title = {Learning to Sense Sparse Signals: Simultaneous Sensing Matrix and Sparsifying Dictionary Optimization},

year = {2008}

}

### OpenURL

### Abstract

Abstract- Sparse signals representation, analysis, and sensing, has received a lot of attention in recent years from the signal processing, optimization, and learning communities. On one hand, the learning of overcomplete dictionaries that facilitate a sparse representation of the image as a liner combination of a few atoms from such dictionary, leads to state-of-the-art results in image and video restoration and image classification. On the other hand, the framework of compressed sensing (CS) has shown that sparse signals can be recovered from far less samples than those required by the classical Shannon-Nyquist Theorem. The goal of this paper is to present a framework that unifies the learning of overcomplete dictionaries for sparse image representation with the concepts of signal recovery from very few samples put forward by the CS theory. The samples used in CS correspond to linear projections defined by a sampling projection matrix. It has been shown that, for example, a non-adaptive random sampling matrix satisfies the fundamental theoretical requirements of CS, enjoying the additional benefit of universality. On the other hand, a projection sensing matrix that is optimally designed for a certain signal class can further improve the reconstruction accuracy or further reduce the necessary number of samples. In this work we introduce a framework for the joint design and optimization, from a set of training images, of the

### Citations

1716 | Compressed sensing
- Donoho
- 2005
(Show Context)
Citation Context ...imension considerably lower than the number of samples required by the ShannonNyquist Theorem (in the order of 2-3 times the sparsity of the signal, regardless of the actual signal bandwidth) [1]-[5],=-=[7]-=-. In addition, signals that are well approximated by sparse representations (i.e., compressible), such as natural images [12]-[18], can be also sensed by linear measurements at a much lower rate than ... |

1652 | Atomic decomposition by basis pursuit
- Chen, Donoho, et al.
- 2001
(Show Context)
Citation Context ...he related definition of mutual coherence of a dictionary, which plays an important role in the success of basis pursuit and the greedy sparsifying orthogonal matching pursuit algorithm as well [26]- =-=[31]-=-; see below). CS deals with the case of low coherence between the sensing and sparsifying matrices. Intuitively, one can see that ... |

1298 | Robust uncertainty principles: exact signal reconstruction from highly incomplete frequency information
- Candes, Romberg, et al.
- 2006
(Show Context)
Citation Context ...ges can be compressed on an appropriate basis, sampling the scene into millions of pixels to obtain high resolution images that are then to be compressed before processing, seems often to be wasteful =-=[1]-=--[11]. The main reason why signals in general and images in particular have been traditionally sensed using a large number of samples is the Shannon-Nyquist Theorem: the sampling rate must be at least... |

1048 | Matching pursuits with time-frequency dictionaries - Mallat, Zhang - 1993 |

655 | Decoding by linear programming - Candes, Tao - 2005 |

634 |
An introduction to compressive sampling
- Candès, Wakin
- 2008
(Show Context)
Citation Context ...l images [12]-[18], can be also sensed by linear measurements at a much lower rate than double their actual bandwidth, as required by the Shannon-Nyquist Theorem, with minimum loss of information [1]-=-=[3]-=-. This means that instead of sensing an image using millions of pixels to obtain high resolution, the image can be sensed directly in compressed form, by sampling a relatively small number of projecti... |

589 | Sparse coding with an overcomplete basis set: A strategy employed by v1 - Olshausen, Field - 1997 |

568 | 2001. A database of human segmented natural images and its application to evaluating segmentation algorithms and measing ecological statistics
- Martin, Fowlkes, et al.
(Show Context)
Citation Context ...ely. The training data consists of 6600 8×8 patches obtained by extracting at random 15 patches from each one of the 440 images in the training set (250 images from the Berkeley segmentation data set =-=[47]-=- and 190 images from the Labelme data set [48]). The testing data consists of 120000 patches corresponding to all the non-overlapping patches of size 8×8 extracted from the remaining 50 images in the ... |

525 | Greed is good: Algorithmic results for sparse approximation
- Tropp
- 2004
(Show Context)
Citation Context ...ich is the largest absolute normalized inner product between the atoms of the dictionary (see Equation (7) in the next section for the exact definition). If the following inequality holds [26], [27], =-=[30]-=-, ‖... |

407 |
The K-SVD, an algorithm for designing overcomplete dictionaries for sparse representation
- Aharon, Elad, et al.
(Show Context)
Citation Context ... PROCESSING 7 incoherence and accuracy of the reconstructed images. In Section III, we review the KSVD algorithm for learning overcomplete non-parametric dictionaries from image datasets, [12], [18], =-=[42]-=-, and introduce the novel coupled-KSVD as a necessary modification to include the simultaneous learning of both the dictionary and the corresponding sampling projection matrix. In Section IV, we prese... |

365 | Elad.“Optimally sparse representation in general (nonorthogonal) dictionaries via 1 minimization
- Donoho, Michael
- 2003
(Show Context)
Citation Context ...as possible. A measure of coherence between ... |

350 | Orthogonal matching pursuit: recursive function approximation with applications to wavelet decomposition
- Pati, Rezaiifar, et al.
- 1993
(Show Context)
Citation Context ...edy algorithm used to solve sparse representation problems, is guaranteed to succeed in finding the correct ... |

318 | Robust face recognition via sparse representation - Wright, Yang, et al. - 2009 |

240 | 80 million tiny images: A large data set for nonparametric object and scene recognition
- Torralba, Fergus, et al.
- 2008
(Show Context)
Citation Context ...loped is based on image patches, as commonly exploited in image processing. While in principle we could work with entire images, this is computationally unfeasible. For tiny 32 × 32 images, following =-=[51]-=-, we obtain results consistent with the work reported above for the patches, see Figure 10. Of course, images are much larger than this, and algorithms of the type of KSVD as here developed, or basica... |

214 |
Sparse representations in unions of bases
- Gribonval, Nielsen
(Show Context)
Citation Context ...sible. A measure of coherence between ... |

199 | D.: Statistics of Natural Images and Models
- Huang, Mumford
- 1999
(Show Context)
Citation Context ...e, to sample specific classes of sparse signals, and in particular, natural images. Considering well-known characteristics and models for the second order statistics of natural images, see e.g., [43]-=-=[46]-=-, Weiss et al., [38], first showed that the Signal to Noise Ratio (SNR) of images projected using (almost any) random sampling matrices goes to zero as the number of pixels increase, while itIEEE TRA... |

184 | Self-taught learning: Transfer learning from unlabeled data
- Raina, Battle, et al.
- 2007
(Show Context)
Citation Context ...se representations form the basis of many successful image processing and analysis algorithms, from JPEG and JPEG2000 compression [19]-[21], to image enhancement and classification, e.g., [12], [22]- =-=[25]-=-. • Incoherent Sampling: Let ... |

147 | Signal recovery from partial information via orthogonal matching pursuit
- Tropp, Gilbert
(Show Context)
Citation Context ...lgorithm used to solve sparse representation problems, is guaranteed to succeed in finding the correct ... |

145 | Sparse geometrical image representations with bandelets - Pennec, Mallat - 2003 |

142 | Singlepixel imaging via compressive sampling
- Duarte, Davenport, et al.
- 2008
(Show Context)
Citation Context ... fixed sparsifying basis ... |

138 | Statistical models for images: compression restoration and synthesis - Simoncelli - 1997 |

133 | Infomax and maximum likelihood for blind source separation
- Cardoso
- 2002
(Show Context)
Citation Context ... ... |

108 | Unsupervised learning of invariant feature hierarchies with applications to object recognition - Ranzato, Huang, et al. - 2007 |

106 | Sparse Representation for Color Image Restoration
- Mairal, Eldar, et al.
- 2008
(Show Context)
Citation Context ... sparsity of the signal, regardless of the actual signal bandwidth) [1]-[5],[7]. In addition, signals that are well approximated by sparse representations (i.e., compressible), such as natural images =-=[12]-=--[18], can be also sensed by linear measurements at a much lower rate than double their actual bandwidth, as required by the Shannon-Nyquist Theorem, with minimum loss of information [1]-[3]. This mea... |

103 | Uniform uncertainty principle and signal recovery via regularized orthogonal matching pursuit
- Needell, Vershynin
- 2009
(Show Context)
Citation Context ...), i.e., incoherence between ... |

100 | On advances in statistical modeling of natural images - Srivastava, Lee, et al. |

97 | a Wavelet Tour of Signal Processing 2 nd Ed - Mallat - 1999 |

89 | Deterministic constructions of compressed sensing matrices
- DeVore
- 2007
(Show Context)
Citation Context ...s just optimized for the signal class. Theoretical studies, with practical implications, regarding the construction of deterministic sampling matrices and their RIP, are starting to appear, e.g., see =-=[40]-=-, [41] and references therein. In this paper, we introduce a framework for simultaneously learning the overcomplete non-parametric dictionary ... |

88 |
Discriminative learned dictionaries for local image analysis
- Mairal, Bach, et al.
- 2008
(Show Context)
Citation Context .... Sparse representations form the basis of many successful image processing and analysis algorithms, from JPEG and JPEG2000 compression [19]-[21], to image enhancement and classification, e.g., [12], =-=[22]-=-- [25]. • Incoherent Sampling: Let ... |

76 | Stochastic models for generic images - Mumford, Gidas |

75 | Compressed sensing and redundant dictionaries
- Rauhut, Schnass, et al.
- 2008
(Show Context)
Citation Context ... regression.IEEE TRANSACTIONS ON IMAGE PROCESSING 5 notion in CS theory that comes to the rescue in this scenario is the Restricted Isometry Property (RIP) [1]- [4] (and references therein, see also =-=[34]-=- for some results for overcomplete dictionaries). The S-restricted isometry constant is the smallest 0 <... |

67 | What makes a good model of natural images - Weiss, Freeman |

61 | An overview of JPEG-2000
- Marcellin, Gormish, et al.
(Show Context)
Citation Context ...wavelets, sinusoids, or a learned (overcomplete) dictionary. Sparse representations form the basis of many successful image processing and analysis algorithms, from JPEG and JPEG2000 compression [19]-=-=[21]-=-, to image enhancement and classification, e.g., [12], [22]- [25]. • Incoherent Sampling: Let ... |

51 | Learning multiscale sparse representations for image and video restoration. SIAM Multiscale Modeling and Simulation
- Mairal, Sapiro, et al.
(Show Context)
Citation Context ...ly impossible. On the other hand, following once again the state of the art results for image enhancement via KSVD, we should work with overlapping patches (e.g., 8×8 or multiscale up to 20 × 20, see =-=[52]-=-). Ideally, we would like then to have the dictionary acting on all the overlapping ... |

50 | Framing pyramids - Do, Vetterli - 2003 |

47 | Image denoising via learned dictionaries and sparse representation
- Elad, Aharon
- 2006
(Show Context)
Citation Context ...sity of the signal, regardless of the actual signal bandwidth) [1]-[5],[7]. In addition, signals that are well approximated by sparse representations (i.e., compressible), such as natural images [12]-=-=[18]-=-, can be also sensed by linear measurements at a much lower rate than double their actual bandwidth, as required by the Shannon-Nyquist Theorem, with minimum loss of information [1]-[3]. This means th... |

40 | Imaging via compressive sampling
- Romberg
- 2008
(Show Context)
Citation Context ...of dimension considerably lower than the number of samples required by the ShannonNyquist Theorem (in the order of 2-3 times the sparsity of the signal, regardless of the actual signal bandwidth) [1]-=-=[5]-=-,[7]. In addition, signals that are well approximated by sparse representations (i.e., compressible), such as natural images [12]-[18], can be also sensed by linear measurements at a much lower rate t... |

34 | Optimized projections for compressed sensing
- Elad
- 2007
(Show Context)
Citation Context ...are not orthonormal bases and hence, the full extent of the CS theoryIEEE TRANSACTIONS ON IMAGE PROCESSING 6 does not entirely apply here (see for example [7], [34] for some results). Recently, Elad =-=[37]-=- (see also [38], [39]), showed experimentally that a well-designed sampling matrix can significantly improve the performance of CS when compared to random matrices, in terms of improving the incoheren... |

27 | Chirp sensing codes: Deterministic compressed sensing measurements for fast recovery. Applied and Computational Harmonic Analysis
- Howard, Searle, et al.
- 2009
(Show Context)
Citation Context ...ases and hence, the full extent of the CS theoryIEEE TRANSACTIONS ON IMAGE PROCESSING 6 does not entirely apply here (see for example [7], [34] for some results). Recently, Elad [37] (see also [38], =-=[39]-=-), showed experimentally that a well-designed sampling matrix can significantly improve the performance of CS when compared to random matrices, in terms of improving the incoherence for a given dictio... |

16 | Frame based signal compression using method of optimal directions (MOD - Engan, Aase, et al. - 1999 |

16 |
Compression of facial images using the K-SVD algorithm
- Bryt, Elad
- 2008
(Show Context)
Citation Context ...ible. A key result in image processing is that images can be coded and sparsely represented more efficiently using (often learned) overcomplete dictionaries rather than fixed bases, e.g., [18], [33], =-=[36]-=-. Let ... |

16 | A negative result concerning explicit matrices with the restricted isometry property, Tech. report, Rice DSP website, 2008, Available at http://dsp.rice.edu/ cs
- Chandar
(Show Context)
Citation Context ... optimized for the signal class. Theoretical studies, with practical implications, regarding the construction of deterministic sampling matrices and their RIP, are starting to appear, e.g., see [40], =-=[41]-=- and references therein. In this paper, we introduce a framework for simultaneously learning the overcomplete non-parametric dictionary ... |

15 | Learning compressed sensing
- Weiss, Chang, et al.
- 2007
(Show Context)
Citation Context ...atrix ... |

10 |
Compressed Sensing and Best k-term Approximation. Preprint, http://www.math.sc.edu/∼devore/publications/CDDSensing 6.pdf
- Cohen, Dahmen, et al.
- 2006
(Show Context)
Citation Context ...can be compressed on an appropriate basis, sampling the scene into millions of pixels to obtain high resolution images that are then to be compressed before processing, seems often to be wasteful [1]-=-=[11]-=-. The main reason why signals in general and images in particular have been traditionally sensed using a large number of samples is the Shannon-Nyquist Theorem: the sampling rate must be at least twic... |

8 | Compressive sampling vs. conventional imaging - Haupt, Nowak - 2006 |

4 | K.: Sparse image coding using learned overcomplete dictionaries
- Murray, Kreutz-Delgado
- 2004
(Show Context)
Citation Context ...s possible. A key result in image processing is that images can be coded and sparsely represented more efficiently using (often learned) overcomplete dictionaries rather than fixed bases, e.g., [18], =-=[33]-=-, [36]. Let ... |

4 |
Labelme: the open annotation tool
- Torralba, Murphy, et al.
- 2007
(Show Context)
Citation Context ...tches obtained by extracting at random 15 patches from each one of the 440 images in the training set (250 images from the Berkeley segmentation data set [47] and 190 images from the Labelme data set =-=[48]-=-). The testing data consists of 120000 patches corresponding to all the non-overlapping patches of size 8×8 extracted from the remaining 50 images in the Berkeley dataset that are not in the training ... |

2 |
Sparse modeling of textures. Preprint Ceremade
- Peyre
- 2007
(Show Context)
Citation Context ...e �... |

1 |
basis compressed sensing,” Preprint Ceremade 2007-20. [Online]. Available: http://www.ceremade.dauphine.fr/~peyre/publications/07-Preprint-Peyre-BestBasisCS.pdf
- Peyré, “Best
(Show Context)
Citation Context ...h more classical compressed sensing scenarios where random sensing matrices are used. The same framework can be used to learn the sparsifying dictionary while keeping the sensing matrix fix (see also =-=[50]-=-). As mentioned in the introduction, the theoretical results for CS support the use of ℓ�optimization, while KSVD-type of algorithms have traditionally been based on OMP (for which the results are wea... |