## Learning multiscale sparse representations for image and video restoration (2007)

### Cached

### Download Links

Citations: | 51 - 17 self |

### BibTeX

@TECHREPORT{Mairal07learningmultiscale,

author = {Julien Mairal and Guillermo Sapiro and Michael Elad},

title = {Learning multiscale sparse representations for image and video restoration},

institution = {},

year = {2007}

}

### OpenURL

### Abstract

Abstract. This paper presents a framework for learning multiscale sparse representations of color images and video with overcomplete dictionaries. A single-scale K-SVD algorithm was introduced in [1], formulating sparse dictionary learning for grayscale image representation as an optimization problem, efficiently solved via Orthogonal Matching Pursuit (OMP) and Singular Value Decomposition (SVD). Following this work, we propose a multiscale learned representation, obtained by using an efficient quadtree decomposition of the learned dictionary, and overlapping image patches. The proposed framework provides an alternative to pre-defined dictionaries such as wavelets, and shown to lead to state-of-the-art results in a number of image and video enhancement and restoration applications. This paper describes the proposed framework, and accompanies it by numerous examples demonstrating its strength. Key words. Image and video processing, sparsity, dictionary, multiscale representation, denoising, inpainting, interpolation, learning. AMS subject classifications. 49M27, 62H35

### Citations

2027 | A Wavelet Tour of Signal Processing - Mallat - 1999 |

1652 | Atomic decomposition by basis pursuit
- Chen, Donoho, et al.
- 2001
(Show Context)
Citation Context ...f Equation (2.2), as the non-convexity of the functional we are considering makes this problem difficult (in fact, NP-Hard) in general. A well-known and alternative approach is the Basis Pursuit (BP) =-=[7]-=-, which suggests a convexification of the problem by using the ℓ1-norm instead of ℓ0 . Nevertheless, when working with small patches, greedy algorithms prove to be far more efficient. • Dictionary Upd... |

847 | The design and use of steerable filters
- Freeman, Adelson
- 1991
(Show Context)
Citation Context ...a sparsity criterion has been a major challenge in recent years. Such attempts include wavelets [26], curvelets [5, 6], contourlets [14, 15], wedgelets [16], bandlets [27, 28], and steerable wavelets =-=[20, 38]-=-. These methods lead to many effective algorithms in image processing, e.g., image denoising [34]. In this paper, instead of designing the best pre-defined dictionary for image reconstruction, we prop... |

601 |
Numerical methods for least squares problems
- Bjorck
- 1996
(Show Context)
Citation Context ...For the scale s, we denote their index as p among the 4s possible shifts. ThesLEARNING SPARSE AND MULTISCALE REPRESENTATIONS 9 OMP is implemented efficiently using the Modified Gram-Schmidt algorithm =-=[3]-=-. During each selection procedure of the OMP, a scale s, a position p, and an atom ˆ dsl are chosen. For each “root” patch, this step can be achieved in O(( �N−1 s=0 ks)n�ˆα�0) operations. • Dictionar... |

568 | 2001. A database of human segmented natural images and its application to evaluating segmentation algorithms and measing ecological statistics
- Martin, Fowlkes, et al.
(Show Context)
Citation Context ...to the color denoising K-SVD to show that it can compete and provide again state-of-the-art results. Like in [25], we use a data set composed of natural images from the Berkeley Segmentation Database =-=[30]-=-, see Figure 6.7. Numerical results are presented on Table 6.4 and some visual results in Figure 6.8. All the numbers presented here are averaged over 5 experiments for each image and each level of no... |

525 | Greed is good: Algorithmic results for sparse approximation
- Tropp
- 2004
(Show Context)
Citation Context ...gorithm are (refer to Figure 2.1): • Sparse Coding Step: This is performed with an Orthogonal Matching Pursuit (OMP) [12, 13, 29], which proves to be very efficient for diverse approximation problems =-=[17, 40, 41]-=-. The approximation stops when the residual reaches a sphere of radius √ nCσ representing the probability distribution of the noise (C being a constant). More on this can be found in [25]. • Dictionar... |

427 | Shiftable multi-scale transforms
- Simoncelli, Freeman, et al.
- 1992
(Show Context)
Citation Context ...a sparsity criterion has been a major challenge in recent years. Such attempts include wavelets [26], curvelets [5, 6], contourlets [14, 15], wedgelets [16], bandlets [27, 28], and steerable wavelets =-=[20, 38]-=-. These methods lead to many effective algorithms in image processing, e.g., image denoising [34]. In this paper, instead of designing the best pre-defined dictionary for image reconstruction, we prop... |

407 |
The K-SVD, an algorithm for designing overcomplete dictionaries for sparse representation
- Aharon, Elad, et al.
(Show Context)
Citation Context ...D § Abstract. This paper presents a framework for learning multiscale sparse representations of color images and video with overcomplete dictionaries. A single-scale K-SVD algorithm was introduced in =-=[1]-=-, formulating sparse dictionary learning for grayscale image representation as an optimization problem, efficiently solved via Orthogonal Matching Pursuit (OMP) and Singular Value Decomposition (SVD).... |

373 | Image inpainting
- Bertalmoi, Sapiro, et al.
(Show Context)
Citation Context ...matrix whose size is the same as El and where each column corresponding to an index [i, j] isRijβ. This problem is known as a weighted rank-one approximation matrix (see [40]). Inpainting (see, e.g., =-=[2, 9]-=-) consists of filling in holes in images. Within the limits of our model, it becomes possible to address a particular case of inpainting. By considering small random holes as areas with infinite power... |

351 | Image denoising using scale mixtures of gaussians in the wavelet domain
- Portilla, Strela, et al.
- 2003
(Show Context)
Citation Context ... curvelets [5, 6], contourlets [14, 15], wedgelets [16], bandlets [27, 28], and steerable wavelets [20, 38]. These methods lead to many effective algorithms in image processing, e.g., image denoising =-=[34]-=-. In this paper, instead of designing the best pre-defined dictionary for image reconstruction, we propose to learn it from examples. In [1], the K-SVD algorithm is proposed for learning a single-scal... |

300 | Just relax: Convex programming methods for identifying sparse signals in noise
- Tropp
- 2006
(Show Context)
Citation Context ...gorithm are (refer to Figure 2.1): • Sparse Coding Step: This is performed with an Orthogonal Matching Pursuit (OMP) [12, 13, 29], which proves to be very efficient for diverse approximation problems =-=[17, 40, 41]-=-. The approximation stops when the residual reaches a sphere of radius √ nCσ representing the probability distribution of the noise (C being a constant). More on this can be found in [25]. • Dictionar... |

297 | Stable recovery of sparse overcomplete representations in the presence of noise
- Donoho, Elad, et al.
- 2006
(Show Context)
Citation Context ...gorithm are (refer to Figure 2.1): • Sparse Coding Step: This is performed with an Orthogonal Matching Pursuit (OMP) [12, 13, 29], which proves to be very efficient for diverse approximation problems =-=[17, 40, 41]-=-. The approximation stops when the residual reaches a sphere of radius √ nCσ representing the probability distribution of the noise (C being a constant). More on this can be found in [25]. • Dictionar... |

268 |
Image denoising via sparse and redundant representations over learned dictionaries
- Elad, Aharon
- 2006
(Show Context)
Citation Context ...UILLERMO SAPIRO, AND MICHAEL ELAD Table 1 Comparison of PSNR results of several denoising algorithms. Each cell is divided into four parts. The top-left cell shows the results from the original K-SVD =-=[20]-=-, and the top-right cell presents the most recent state-of-the-art results [12]. The bottom-left cell is devoted to our results for N =1scale and the bottom-right cell to N =2scales. Each time the bes... |

263 | A review of image denoising algorithms, with a new one, Multiscale Modeling and Simulation 4 (2
- Buades, Coll, et al.
- 2005
(Show Context)
Citation Context ...ks such as [10, 19, 20, 22, 35, 38]. The proposed algorithm also competes favorably with the most recent and state-of-the-art result in this field [12], which is based on the nonlocal means algorithm =-=[4]-=-. Our framework for color image denoising also competes favorably with the best known algorithm in this field [11], and the results for the other presented applications such as color video denoising a... |

226 | Fields of experts: a framework for learning image priors
- Roth, Black
- 2005
(Show Context)
Citation Context ...RAL, G. SAPIRO AND M. ELAD various image and video processing tasks, obtaining results that outperform previous works. Our results for denoising grayscale images outperform for instance works such as =-=[9, 18, 19, 21, 34, 37]-=-. The proposed algorithm also competes favorably with the most recent and state-of-the-art result in this field [11], which is based on the nonlocal means algorithm [4]. Our framework for color image ... |

218 | Image denoising by sparse 3d transform-domain collaborative filtering - Dabov, Foi, et al. - 2007 |

167 | Region filling and object removal by exemplar-based image inpainting
- Criminisi, Perez, et al.
- 2004
(Show Context)
Citation Context ...ve: (di,αi) = arg min �β ⊗ E + (1n×m − β) ⊗ di−1α α,�d�2=1 T i−1 − dα T � 2 F, where 1n×m is an n × m matrix filled with ones. Fig. 2.2. A weighted one-rank approximation algorithm. Inpainting, e.g., =-=[2, 8]-=-, consists of filling-in holes in images. Within the limits of our model, it becomes possible to address a particular case of inpainting. By considering small random holes as areas with infinite power... |

145 | Sparse geometrical image representations with bandelets
- Pennec, Mallat
- 2003
(Show Context)
Citation Context ...iscale dictionary which fulfills a sparsity criterion has been a major challenge in recent years. Such attempts include wavelets [28], curvelets [5, 6], contourlets [15, 16], wedgelets [17], bandlets =-=[24, 29]-=-, and steerable wavelets [21, 39]. These methods lead to many effective algorithms in image processing, e.g., image denoising [35]. In this paper, instead of designing the best predefined dictionary f... |

142 |
Matching pursuit in a time-frequency dictionary
- Mallat, Zhang
- 1993
(Show Context)
Citation Context ...ordinates [i,j] from the image written as a column vector. The main steps of the algorithm are (refer to Figure 2.1): • Sparse Coding Step: This is performed with an Orthogonal Matching Pursuit (OMP) =-=[12, 13, 29]-=-, which proves to be very efficient for diverse approximation problems [17, 40, 41]. The approximation stops when the residual reaches a sphere of radius √ nCσ representing the probability distributio... |

126 |
Wedgelets: Nearly minimax estimation of edges
- Donoho
- 1999
(Show Context)
Citation Context ...gn the best multiscale dictionary which fulfils a sparsity criterion has been a major challenge in recent years. Such attempts include wavelets [26], curvelets [5, 6], contourlets [14, 15], wedgelets =-=[16]-=-, bandlets [27, 28], and steerable wavelets [20, 38]. These methods lead to many effective algorithms in image processing, e.g., image denoising [34]. In this paper, instead of designing the best pre-... |

124 |
Weighted low-rank approximations
- Srebro, Jaakkola
- 2003
(Show Context)
Citation Context ...SENTATIONS 5 where β l is the matrix whose size is the same as El and where each column corresponding to an index [i,j] is Rijβ. This problem is known as a weighted one-rank approximation matrix (see =-=[39]-=-). The algorithm that we used to approximate a solution is presented Figure 2.2. Input: E (n×m matrix), β (n×m matrix of weights), iter (number of iterations, typically 5). Output: diter (n × 1 vector... |

116 | Efficient Learning of Sparse Representations with an Energy-Based Model - Ranzato, Poultney, et al. - 2006 |

114 | Adaptive greedy approximations
- Davis, Mallat, et al.
- 1997
(Show Context)
Citation Context ..., forming the output patch as a column vector. The main steps of the algorithm are (refer to Figure 1) the following: • Sparse Coding step: This is performed with an orthogonal matching pursuit (OMP) =-=[13, 14, 30]-=-, a greedy algorithm that proves to be very efficient for diverse approximation problems [18, 41, 42]. The approximation stops when the residual reaches a sphere of radius √ nCσ representing the proba... |

106 | Sparse Representation for Color Image Restoration
- Mairal, Eldar, et al.
- 2008
(Show Context)
Citation Context ...rseland model suggests that such dictionaries exist for various classes of signals, and that the sparsity of a signal decomposition is a powerful model in many image and video processing applications =-=[1, 19, 25, 35]-=-. An important assumption, commonly and successfully used in image processing, is the existence of multiscale features in images. Attempting to design the best multiscale dictionary which fulfils a sp... |

91 | Demosaicing: image reconstruction from color ccd samples
- KIMMEL
- 1999
(Show Context)
Citation Context ...del, we were able to fill-in relatively small holes in the images and we presented state-of-the-art results for image demosaicing, outperforming every specialized interpolation-based methods, such as =-=[7, 22, 23, 31]-=-. Let us now consider the case where w is a white Gaussian noise with a different standard deviation σp > 0 at each location p. Assuming these standard deviations are known, we introduced a vector β c... |

75 |
Adaptive time-frequency decompositions
- Davis, Mallat, et al.
- 1994
(Show Context)
Citation Context ...ordinates [i,j] from the image written as a column vector. The main steps of the algorithm are (refer to Figure 2.1): • Sparse Coding Step: This is performed with an Orthogonal Matching Pursuit (OMP) =-=[12, 13, 29]-=-, which proves to be very efficient for diverse approximation problems [17, 40, 41]. The approximation stops when the residual reaches a sphere of radius √ nCσ representing the probability distributio... |

68 | Optimal Spatial Adaptation for Patch-Based Image Denoising
- Kervrann, Bourlanger
- 2006
(Show Context)
Citation Context ...RAL, G. SAPIRO AND M. ELAD various image and video processing tasks, obtaining results that outperform previous works. Our results for denoising grayscale images outperform for instance works such as =-=[9, 18, 19, 21, 34, 37]-=-. The proposed algorithm also competes favorably with the most recent and state-of-the-art result in this field [11], which is based on the nonlocal means algorithm [4]. Our framework for color image ... |

67 | What makes a good model of natural images
- Weiss, Freeman
(Show Context)
Citation Context ...thm on the energy minimization method introduced in [19]. This explains the significantly superior performance we obtain. Other results on learning single-scale image dictionaries include for example =-=[36, 37, 42]-=-. The structure of this paper is as follows: In Section 2, we briefly review relevant background: The original K-SVD denoising algorithm [1], the extensions to color image denoising, non-homogeneous n... |

66 | A Wavelet Tour of Signal Processing, second edition - Mallat - 1999 |

50 | Space-time completion of video
- Wexler, Shechtman, et al.
(Show Context)
Citation Context ... use the matrix β exactly the same way as we already did for single images. This handles inpainting of relatively small holes. For addressing the general video inpainting problem, one should refer to =-=[33, 43]-=-. Due to the multiscale nature of the proposed scheme, somewhat larger holes can be treated successfully, compared to the single scale algorithm. 4. Additional algorithmic improvements. We now introdu... |

50 | Framing pyramids
- Do, Vetterli
- 2003
(Show Context)
Citation Context ... Attempting to design the best multiscale dictionary which fulfills a sparsity criterion has been a major challenge in recent years. Such attempts include wavelets [28], curvelets [5, 6], contourlets =-=[15, 16]-=-, wedgelets [17], bandlets [24, 29], and steerable wavelets [21, 39]. These methods lead to many effective algorithms in image processing, e.g., image denoising [35]. In this paper, instead of designi... |

49 | Recovering edges in ill-posed inverse problems optimality of curvelet frames
- Candès, Donoho
(Show Context)
Citation Context ...e features in images. Attempting to design the best multiscale dictionary which fulfils a sparsity criterion has been a major challenge in recent years. Such attempts include wavelets [27], curvelets =-=[5, 6]-=-, contourlets [15, 16], wedgelets [17], bandlets [28, 29], and steerable wavelets [21, 39]. These methods lead to many effective algorithms in image processing, e.g., image denoising [35]. In this pap... |

47 | Image denoising via learned dictionaries and sparse representation
- Elad, Aharon
- 2006
(Show Context)
Citation Context ...RAL, G. SAPIRO AND M. ELAD various image and video processing tasks, obtaining results that outperform previous works. Our results for denoising grayscale images outperform for instance works such as =-=[9, 18, 19, 21, 34, 37]-=-. The proposed algorithm also competes favorably with the most recent and state-of-the-art result in this field [11], which is based on the nonlocal means algorithm [4]. Our framework for color image ... |

34 |
Demosaicing by successive approximation
- Li
- 2005
(Show Context)
Citation Context ...del, we were able to fill-in relatively small holes in the images and we presented state-of-the-art results for image demosaicing, outperforming every specialized interpolation-based methods, such as =-=[7, 22, 23, 31]-=-. Let us now consider the case where w is a white Gaussian noise with a different standard deviation σp > 0 at each location p. Assuming these standard deviations are known, we introduced a vector β c... |

34 |
Learning sparse multiscale image representations. Advances in neural information processing systems
- Sallee, Olshausen
- 2002
(Show Context)
Citation Context ...pplications such as color video denoising and inpainting of small holes in image and video, are also among the best we are aware of. The task of learning a multiscale dictionary has been addressed in =-=[32]-=- in the general context of sparsifying image content. Our approach differs from this work in many ways, including: (i) their training algorithm employs a simple steepest descent while ours uses more e... |

25 | Image sequence denoising via sparse and redundant representations
- Protter, Elad
- 2009
(Show Context)
Citation Context ...rseland model suggests that such dictionaries exist for various classes of signals, and that the sparsity of a signal decomposition is a powerful model in many image and video processing applications =-=[1, 19, 25, 35]-=-. An important assumption, commonly and successfully used in image processing, is the existence of multiscale features in images. Attempting to design the best multiscale dictionary which fulfils a sp... |

21 | Color demosaicing using variance of color differences
- Chung, Chan
- 2006
(Show Context)
Citation Context ...del, we were able to fill-in relatively small holes in the images and we presented state-of-the-art results for image demosaicing, outperforming every specialized interpolation-based methods, such as =-=[7, 22, 23, 31]-=-. Let us now consider the case where w is a white Gaussian noise with a different standard deviation σp > 0 at each location p. Assuming these standard deviations are known, we introduced a vector β c... |

20 | Demosaicing using optimal recovery
- Muresan, Parks
- 2005
(Show Context)
Citation Context |

15 | Statistical modeling of images with fields of Gaussian scale mixtures
- Lyu, Simoncelli
- 2007
(Show Context)
Citation Context ...eraged over 5 experiments for each image and each level of noise, to cope with the variability of the PSNR with the different noise realizations. We also compared our results with a very recent paper =-=[24]-=-, which is an extension of [34] with noticeable improvements. In this work, the authors presented some experiments over a data set that has five images in common with the one we chose (house, peppers,... |

15 | Bandelet image approximation and compression
- Mallat, Pennec
- 2005
(Show Context)
Citation Context ...tiscale dictionary which fulfils a sparsity criterion has been a major challenge in recent years. Such attempts include wavelets [26], curvelets [5, 6], contourlets [14, 15], wedgelets [16], bandlets =-=[27, 28]-=-, and steerable wavelets [20, 38]. These methods lead to many effective algorithms in image processing, e.g., image denoising [34]. In this paper, instead of designing the best pre-defined dictionary ... |

15 | talmío, "Video Inpainting Under Constrained Camera Motion
- Patwardhan, Sapiro, et al.
(Show Context)
Citation Context ... use the matrix β exactly the same way as we already did for single images. This handles inpainting of relatively small holes. For addressing the general video inpainting problem, one should refer to =-=[33, 43]-=-. Due to the multiscale nature of the proposed scheme, somewhat larger holes can be treated successfully, compared to the single scale algorithm. 4. Additional algorithmic improvements. We now introdu... |

13 | Image denoising with block-matching and 3D filtering
- Dabov, Foi, et al.
(Show Context)
Citation Context |

11 |
Beyond wavelets
- Do, Vertelli
- 2003
(Show Context)
Citation Context .... Attempting to design the best multiscale dictionary which fulfils a sparsity criterion has been a major challenge in recent years. Such attempts include wavelets [26], curvelets [5, 6], contourlets =-=[14, 15]-=-, wedgelets [16], bandlets [27, 28], and steerable wavelets [20, 38]. These methods lead to many effective algorithms in image processing, e.g., image denoising [34]. In this paper, instead of designi... |

7 |
New tight frames of curvelets and the problem of approximating piecewise C 2 images with piecewise C 2 edges
- Candes, Donoho
(Show Context)
Citation Context ... features in images. Attempting to design the best multiscale dictionary which fulfills a sparsity criterion has been a major challenge in recent years. Such attempts include wavelets [28], curvelets =-=[5, 6]-=-, contourlets [15, 16], wedgelets [17], bandlets [24, 29], and steerable wavelets [21, 39]. These methods lead to many effective algorithms in image processing, e.g., image denoising [35]. In this pap... |

6 |
Bandelet image approximation and compression, Multiscale Model
- Pennec, Mallat
(Show Context)
Citation Context ...iscale dictionary which fulfills a sparsity criterion has been a major challenge in recent years. Such attempts include wavelets [28], curvelets [5, 6], contourlets [15, 16], wedgelets [17], bandlets =-=[24, 29]-=-, and steerable wavelets [21, 39]. These methods lead to many effective algorithms in image processing, e.g., image denoising [35]. In this paper, instead of designing the best predefined dictionary f... |

4 |
geometric image representations with bandelets
- “Sparse
- 2005
(Show Context)
Citation Context ...tiscale dictionary which fulfils a sparsity criterion has been a major challenge in recent years. Such attempts include wavelets [26], curvelets [5, 6], contourlets [14, 15], wedgelets [16], bandlets =-=[27, 28]-=-, and steerable wavelets [20, 38]. These methods lead to many effective algorithms in image processing, e.g., image denoising [34]. In this paper, instead of designing the best pre-defined dictionary ... |

3 |
Color image denoising by sparse 3d collaborative filtering with grouping constraint in luminance-chrominance space
- Dabov, Foi, et al.
- 2007
(Show Context)
Citation Context ...e-of-the-art result in this field [12], which is based on the nonlocal means algorithm [4]. Our framework for color image denoising also competes favorably with the best known algorithm in this field =-=[11]-=-, and the results for the other presented applications such as color video denoising and inpainting of small holes in image and video are also among the best we are aware of. The task of learning a mu... |

2 |
tight frames of curvelets and the problem of approximating piecewise C 2 images with piecewise C 2 edges
- New
- 2004
(Show Context)
Citation Context ...e features in images. Attempting to design the best multiscale dictionary which fulfils a sparsity criterion has been a major challenge in recent years. Such attempts include wavelets [27], curvelets =-=[5, 6]-=-, contourlets [15, 16], wedgelets [17], bandlets [28, 29], and steerable wavelets [21, 39]. These methods lead to many effective algorithms in image processing, e.g., image denoising [35]. In this pap... |

2 |
denoising via sparse and redundant representations over learned dictionaries
- Image
(Show Context)
Citation Context ...e image barbara can be observed through the different scales. 3 The results in [11] are the best known denoising results at the time of writing this paper. These go beyond the performance reported in =-=[18, 19, 21, 34]-=-, which until recently were the leading ones, each in its short period of time.sLEARNING SPARSE AND MULTISCALE REPRESENTATIONS 13 With N > 3, our multiscale scheme proves not to be flexible enough to ... |

1 |
image denoising by sparse 3d collaborative filtering with grouping constraint in luminance-chrominance space
- Color
- 2007
(Show Context)
Citation Context ...e-of-the-art result in this field [11], which is based on the nonlocal means algorithm [4]. Our framework for color image denoising also competes favorably with the best known algorithm in this field =-=[10]-=-, and the results for the other presented applications such as color video denoising and inpainting of small holes in image and video, are also among the best we are aware of. The task of learning a m... |