## Generalizing the non-local-means to super-resolution reconstruction (2009)

### Cached

### Download Links

Venue: | IN IEEE TRANSACTIONS ON IMAGE PROCESSING |

Citations: | 34 - 4 self |

### BibTeX

@INPROCEEDINGS{Protter09generalizingthe,

author = {Matan Protter and Michael Elad and Hiroyuki Takeda and Peyman Milanfar},

title = {Generalizing the non-local-means to super-resolution reconstruction },

booktitle = {IN IEEE TRANSACTIONS ON IMAGE PROCESSING},

year = {2009},

pages = {36},

publisher = {}

}

### OpenURL

### Abstract

Super-resolution reconstruction proposes a fusion of several low-quality images into one higher quality result with better optical resolution. Classic super-resolution techniques strongly rely on the availability of accurate motion estimation for this fusion task. When the motion is estimated inaccurately, as often happens for nonglobal motion fields, annoying artifacts appear in the super-resolved outcome. Encouraged by recent developments on the video denoising problem, where state-of-the-art algorithms are formed with no explicit motion estimation, we seek a super-resolution algorithm of similar nature that will allow processing sequences with general motion patterns. In this paper, we base our solution on the Nonlocal-Means (NLM) algorithm. We show how this denoising method is generalized to become a relatively simple super-resolution algorithm with no explicit motion estimation. Results on several test movies show that the proposed method is very successful in providing super-resolution on general sequences.

### Citations

2057 | Rapid object detection using a boosted cascade of simple features
- Viola, Jones
- 2001
(Show Context)
Citation Context ...block difference can be computed using a small constant number of calculations per pixel, regardless of block size. Using such a patch structure has only a slight effect on the quality of the outputs =-=[48]-=-. Authorized licensed use limited to: IEEE Xplore. Downloaded on December 18, 2008 at 18:02 from IEEE Xplore. Restrictions apply.sPROTTER et al.: GENERALIZING THE NONLOCAL-MEANS TO SUPER-RESOLUTION RE... |

1374 |
Nonlinear total variation based noise removal algorithms
- Rudin, Osher, et al.
- 1992
(Show Context)
Citation Context ...he imaging process, bringing to the same resolution as the input sequence . The additional TV (Total Variation) expression comes to regularize the deblurring that should take place in this expression =-=[38]-=-, forcing piece-wise smoothness of the desired image, by accumulating the norms of the gradients with norm. Taking a close look at the penalty term we have just written reveals that it cannot provide ... |

783 | Optimization by Vector Space Methods
- Luenberger
- 1969
(Show Context)
Citation Context ...outcome. While solving the above equation directly is possible in principle, it requires an inversion of a very large matrix. Instead, we adopt an iterative approach based on the fixed-point strategy =-=[36]-=-, [37]. Denoting the outcome of the previous iteration, and the desired outcome of the current iteration, we rewrite (7) with assignments of iteration stage per each instance of the unknown . The equa... |

665 | Bilateral filtering for gray and color images - Tomasi, Manduchi - 1998 |

395 |
Digital Image Warping
- Wolberg
- 1994
(Show Context)
Citation Context ...n of being a scaled-up version of the reference frame we aim to super-resolve. This scale-up is done using a conventional image interpolation algorithm such as bilinear, bicubic or the Lanczos method =-=[45]-=-, [46]. The interpolated images are only crude estimates of the desired outcome, and, therefore, the weights computed by relying on these estimated are also somewhat crude. Since after one iteration o... |

305 | Limits on super-resolution and how to break them
- Baker, Kanade
- 2002
(Show Context)
Citation Context ... the motion in the sequence, followed by a fusion of the inputs according to these motion vectors. A wide variety of super resolution algorithms have been developed in the past 2 decades; we refer to =-=[1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26]-=- as representatives of this vast literature. In the currently available super-resolution algorithms, only global motion estimation (e.g. 2translation or affine global warp) is accurate enough to lead... |

263 | A review of image denoising algorithms, with a new one, Multiscale Modeling and Simulation 4 (2
- Buades, Coll, et al.
- 2005
(Show Context)
Citation Context ...weakest among the recent motion-estimation-free video denoising algorithms, and yet, it is also the simplest. As such, it stands as a good candidate for generalization. The NLM is posed originally in =-=[31]-=- as a single image denoising method, generalizing the well-known bilateral filter [32], [33]. Denoising is obtained by replacing every pixel with a weighted average of its neighborhood. The weights fo... |

261 |
Improving resolution by image registration
- Irani, Peleg
- 1991
(Show Context)
Citation Context ...lore. Restrictions apply.s42 IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 18, NO. 1, JANUARY 2009 Fig. 4. Relation between the patch sizes � and � a@�0 IA� CIdemonstrated on specific examples: ‘�Y �“ a=-=[3,1]-=-, [3,2], and [3,3]. Fig. 5. Description of the expression h ‚ rˆ 0 ‚ � in (21). Since there are two different scales (and sizes) of patches used, we also must differentiate between the patch extractio... |

234 | An EM algorithm for wavelet-based image restoration
- Figueiredo, Nowak
- 2003
(Show Context)
Citation Context ...second part is a classic simple deblurring problem. This is of course an underdetermined problem (since H is usually singular), and needs regularization. We will not discuss how to solve it here, see =-=[39, 40, 41, 42, 43, 44]-=- for a background view of this field, along with several state-of-the-art deblurring methods. The idea of breaking the super-resolution task into two parts – fusing the inputs and then deblurring – ha... |

210 | Extraction of highresolution frames from video sequences
- Schultz, Stevenson
- 1996
(Show Context)
Citation Context ... the motion in the sequence, followed by a fusion of the inputs according to these motion vectors. A wide variety of super resolution algorithms have been developed in the past 2 decades; we refer to =-=[1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26]-=- as representatives of this vast literature. In the currently available super-resolution algorithms, only global motion estimation (e.g. 2translation or affine global warp) is accurate enough to lead... |

207 | Restoration of a single superresolution image from several blurred, noisy, and undersampled measured images
- Elad, Feuer
- 1997
(Show Context)
Citation Context ... the motion in the sequence, followed by a fusion of the inputs according to these motion vectors. A wide variety of super resolution algorithms have been developed in the past 2 decades; we refer to =-=[1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26]-=- as representatives of this vast literature. In the currently available super-resolution algorithms, only global motion estimation (e.g. 2translation or affine global warp) is accurate enough to lead... |

202 |
Super-resolution image reconstruction: A technical overview
- Park, Park, et al.
- 2003
(Show Context)
Citation Context |

188 |
Multiframe image restoration and registration
- Tsai, Huang
- 1984
(Show Context)
Citation Context ...he motion in the sequence, followed by a fusion of the inputs according to these motion vectors. A wide variety of super resolution algorithms have been developed in the past two decades; we refer to =-=[1]-=-–[26] as representatives of this vast literature. In the currently available super-resolution algorithms, only global motion estimation (e.g., translation or affine global warp) is accurate enough to ... |

164 | Fast and robust multiframe super resolution
- Farsiu, Robinson, et al.
- 2004
(Show Context)
Citation Context ...long with several state-of-the-art deblurring methods. The idea of breaking the super-resolution task into two parts—fusing the inputs and then deblurring—has been suggested previously in [13], [18], =-=[20]-=-, and [21]. In the case of pure translational motion, and when both problems are treated as maximum-likelihood (which is not done here), such a separation is equivalent to the joint solution. It is im... |

157 | Joint map registration and high-resolution image estimation using a sequence of undersampled images
- Hardie, Barnard, et al.
- 1997
(Show Context)
Citation Context |

138 |
On the bilateral filter and ways to improve it
- Elad
(Show Context)
Citation Context ...lso the simplest. As such, it stands as a good candidate for generalization. The NLM is posed originally in [31] as a single image denoising method, generalizing the well-known bilateral filter [32], =-=[33]-=-. Denoising is obtained by replacing every pixel with a weighted average of its neighborhood. The weights for this computation are evaluated by using block-matching fit between image patches centered ... |

128 | Super-resolution Video Reconstruction with Arbitrary Sampling Latices and Nonzero Aperture Time
- Patti, Sezan, et al.
- 1997
(Show Context)
Citation Context |

110 | Kernel Regression for Image Processing and Reconstruction
- Takeda, Farsiu, et al.
- 2007
(Show Context)
Citation Context ...1 As we shall see next, in this framework the coefficients �‘�Y �Y �Y �“ are all restricted to be positive. This is a shortcoming, which can be overcome by extending the framework to higher order—see =-=[34]-=-. Authorized licensed use limited to: IEEE Xplore. Downloaded on December 18, 2008 at 18:02 from IEEE Xplore. Restrictions apply. (1)s38 IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 18, NO. 1, JANUARY ... |

106 |
Recursive reconstruction of high resolution image from noisy undersampled multiframes
- Kim, Bose, et al.
- 1990
(Show Context)
Citation Context ...estrictions apply.s42 IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 18, NO. 1, JANUARY 2009 Fig. 4. Relation between the patch sizes � and � a@�0 IA� CIdemonstrated on specific examples: ‘�Y �“ a[3,1], =-=[3,2]-=-, and [3,3]. Fig. 5. Description of the expression h ‚ rˆ 0 ‚ � in (21). Since there are two different scales (and sizes) of patches used, we also must differentiate between the patch extraction opera... |

85 |
Improved resolution from sub-pixel shifted pictures
- Ur, Gross
- 1992
(Show Context)
Citation Context |

74 |
Nonlinear programing (2 nd ed
- Bertsekas
- 1999
(Show Context)
Citation Context ...e. While solving the above equation directly is possible in principle, it requires an inversion of a very large matrix. Instead, we adopt an iterative approach based on the fixed-point strategy [36], =-=[37]-=-. Denoting the outcome of the previous iteration, and the desired outcome of the current iteration, we rewrite (7) with assignments of iteration stage per each instance of the unknown . The equation w... |

73 |
Iterative methods for image deblurring
- Biemond, Lagendijk, et al.
- 1990
(Show Context)
Citation Context ... second part is a classic simple deblurring problem. This is of course an under-determined problem (since is usually singular), and needs regularization. We will not discuss how to solve it here, see =-=[39]-=-–[44] for a background view of this field, along with several state-of-the-art deblurring methods. The idea of breaking the super-resolution task into two parts—fusing the inputs and then deblurring—h... |

71 | Advances and challenges in super-resolution
- Farsiu, Robinson, et al.
- 2004
(Show Context)
Citation Context ...several state-of-the-art deblurring methods. The idea of breaking the super-resolution task into two parts—fusing the inputs and then deblurring—has been suggested previously in [13], [18], [20], and =-=[21]-=-. In the case of pure translational motion, and when both problems are treated as maximum-likelihood (which is not done here), such a separation is equivalent to the joint solution. It is important to... |

67 | A fast super-resolution reconstruction algorithm for pure translational motion and common spaceinvariant blur
- Elad, Hel-Or
- 2001
(Show Context)
Citation Context ...his field, along with several state-of-the-art deblurring methods. The idea of breaking the super-resolution task into two parts—fusing the inputs and then deblurring—has been suggested previously in =-=[13]-=-, [18], [20], and [21]. In the case of pure translational motion, and when both problems are treated as maximum-likelihood (which is not done here), such a separation is equivalent to the joint soluti... |

66 | Fundamental limits of reconstruction-based superresolution algorithms under local translation
- Lin, Shum
(Show Context)
Citation Context |

64 |
Nonlocal linear image regularization and supervised segmentation. Multiscale Model
- Gilboa, Osher
(Show Context)
Citation Context ... intuitively as algorithmic formulas, as in (1). We claim that both these filters can be derived by minimizing a properly defined penalty function. Following the rationale and steps taken in [33] and =-=[35]-=-, we present such a penalty function, and show how these algorithms emerge from it. This will prove valuable when taking the next step of generalizing these methods to a super-resolution reconstructio... |

62 | Fast image and video denoising via nonlocal means of similar neighborhoods
- Mahmoudi, Sapiro
- 2005
(Show Context)
Citation Context ...duced in order to make the algorithm practical. We will now describe a few speedup methods for the proposed algorithm. Several of the methods to speedup the NLM algorithm were suggested originally in =-=[47]-=-, and were adopted in our simulations. 1) Computing the weights can be done using block differences in the low-resolution images, instead of on the interpolated images. This saves a factor of . 2) Com... |

61 | A frequency domain approach to registration of aliased images with application to super-resolution
- Vandewalle, Süsstrunk, et al.
- 2006
(Show Context)
Citation Context |

54 | Bayesian wavelet-based image deconvolution: A GEM algorithm exploiting a class of heavy-tailed priors
- Bioucas-Dias
- 2006
(Show Context)
Citation Context ...nd part is a classic simple deblurring problem. This is of course an under-determined problem (since is usually singular), and needs regularization. We will not discuss how to solve it here, see [39]–=-=[44]-=- for a background view of this field, along with several state-of-the-art deblurring methods. The idea of breaking the super-resolution task into two parts—fusing the inputs and then deblurring—has be... |

39 | Improving Image Resolution using Subpixel Motion - Pelcg, Keren, et al. - 1987 |

37 | Resolution enhancement of monochrome and color video using motion compensation
- Tom, Katsaggelos
- 2001
(Show Context)
Citation Context |

35 | Superresolution still and video reconstruction from MPEG-coded video
- Altunbasak, Patti, et al.
- 2002
(Show Context)
Citation Context |

30 |
Resolution enhancement of color video sequences
- Shah, Zakhor
- 1999
(Show Context)
Citation Context |

25 | Image sequence denoising via sparse and redundant representations
- Protter, Elad
- 2009
(Show Context)
Citation Context ... sequences with a general motion pattern) is the video denoising application, where several recent contributions demonstrate state-of-the-art results with algorithms that avoid motion estimation [27]–=-=[30]-=-. Among these, we choose to take a closer look at the Nonlocal Means (NLM) algorithm, with the aim to generalize it to perform super-resolution reconstruction. The NLM is the weakest among the recent ... |

24 | Efficient super-resolution and applications to mosaics
- Zomet, Peleg
- 2000
(Show Context)
Citation Context |

22 |
A computationally efficient image superresolution algorithm
- Nguyen, Milanfar, et al.
- 2001
(Show Context)
Citation Context |

22 |
Improving resolution by image registration,” CVGIP: Graphical Models and Image Processing
- Irani, Peleg
- 1991
(Show Context)
Citation Context |

21 | A spatially adaptive nonparametric regression image deblurring
- Katkovnik, Egiazarian, et al.
- 2005
(Show Context)
Citation Context ...second part is a classic simple deblurring problem. This is of course an underdetermined problem (since H is usually singular), and needs regularization. We will not discuss how to solve it here, see =-=[39, 40, 41, 42, 43, 44]-=- for a background view of this field, along with several state-of-the-art deblurring methods. The idea of breaking the super-resolution task into two parts – fusing the inputs and then deblurring – ha... |

18 | Robust shift and add approach to superresolution
- Farsiu, Robinson, et al.
- 2003
(Show Context)
Citation Context ...eld, along with several state-of-the-art deblurring methods. The idea of breaking the super-resolution task into two parts—fusing the inputs and then deblurring—has been suggested previously in [13], =-=[18]-=-, [20], and [21]. In the case of pure translational motion, and when both problems are treated as maximum-likelihood (which is not done here), such a separation is equivalent to the joint solution. It... |

18 | Denoising image sequences does not require motion estimation
- Buades, Coll, et al.
- 2005
(Show Context)
Citation Context ...ocess sequences with a general motion pattern) is the video denoising application, where several recent contributions demonstrate state-of-the-art results with algorithms that avoid motion estimation =-=[27]-=-–[30]. Among these, we choose to take a closer look at the Nonlocal Means (NLM) algorithm, with the aim to generalize it to perform super-resolution reconstruction. The NLM is the weakest among the re... |

16 | Numerical methods for coupled super-resolution
- Chung, Haber, et al.
(Show Context)
Citation Context |

15 |
A MAP approach for joint motion estimation, segmentation, and super resolution
- Shen, Zhang, et al.
- 2007
(Show Context)
Citation Context ...otion in the sequence, followed by a fusion of the inputs according to these motion vectors. A wide variety of super resolution algorithms have been developed in the past two decades; we refer to [1]–=-=[26]-=- as representatives of this vast literature. In the currently available super-resolution algorithms, only global motion estimation (e.g., translation or affine global warp) is accurate enough to lead ... |

14 |
Filters for common resampling tasks,” in Graphics Gems
- Glassner, Turkowski, et al.
- 1990
(Show Context)
Citation Context ...eing a scaled-up version of the reference frame we aim to super-resolve. This scale-up is done using a conventional image interpolation algorithm such as bilinear, bicubic or the Lanczos method [45], =-=[46]-=-. The interpolated images are only crude estimates of the desired outcome, and, therefore, the weights computed by relying on these estimated are also somewhat crude. Since after one iteration of the ... |

11 |
Direct super-resolution and registration using raw CFA images
- Gotoh, Okutomi
- 2004
(Show Context)
Citation Context |

8 | Moving-window varying size 3D transform-based video denoising
- Rusanovskyy, Dabov, et al.
- 2006
(Show Context)
Citation Context ...ead to a more effective noise suppression due to the independence between the noise in each of those patches. Generalization of the NLM approach to video denoising was shown to be very effective [27]–=-=[29]-=-, leading to state-of-the-art results, while leaning strongly on the fuzziness of the estimated motion. These methods’ successes encourage us to seek ways to exercise the same fuzzy motion estimation ... |

8 |
Digital image reconstruction: deblurring and denoising
- Puetter, Gosnell, et al.
(Show Context)
Citation Context ...second part is a classic simple deblurring problem. This is of course an underdetermined problem (since H is usually singular), and needs regularization. We will not discuss how to solve it here, see =-=[39, 40, 41, 42, 43, 44]-=- for a background view of this field, along with several state-of-the-art deblurring methods. The idea of breaking the super-resolution task into two parts – fusing the inputs and then deblurring – ha... |

8 |
Efficient Huber-Markov edgepreserving image restoration
- Pan, Reeves
- 2006
(Show Context)
Citation Context |

6 |
Space-time adaptation for patch based image sequence restoration
- Boulanger, Kervrann, et al.
(Show Context)
Citation Context ...ow this method can be used for video denoising by extending the very same technique to 3-D neighborhoods [27]. An improvement of this technique, considering varying size neighborhoods is suggested in =-=[28]-=-, so as to trade bias versus variance in an attempt to get the best mean-squared-error (MSE). The NLM was proposed intuitively in [27] and [31], and, thus, it is natural to try to extend it to perform... |

3 |
Superresolution ultrasound imaging using back-projected reconstruction
- Clement, Huttunen, et al.
- 2005
(Show Context)
Citation Context |