A new algorithm is proposed for removing large objects from digital images. The challenge is to fill in the hole that is left behind in a visually plausible way. In the past, this problem has been addressed by two classes of algorithms: 1) “texture synthesis” algorithms for generating large image regions from sample textures and 2) “inpainting ” techniques for filling in small image gaps. The former has been demonstrated for “textures”—repeating twodimensional patterns with some stochasticity; the latter focus on linear “structures ” which can be thought of as one-dimensional patterns, such as lines and object contours. This paper presents a novel and efficient algorithm that combines the advantages of these two approaches. We first note that exemplar-based texture synthesis contains the essential process required to replicate both texture and structure; the success of structure propagation, however, is highly dependent on the order in which the filling proceeds. We propose a best-first algorithm in which the confidence in the synthesized pixel values is propagated in a manner similar to the propagation of information in inpainting. The actual color values are computed using exemplar-based synthesis. In this paper, the simultaneous propagation of texture and structure information is achieved by a single, efficient algorithm. Computational efficiency is achieved by a block-based sampling process. A number of examples on real and synthetic images demonstrate the effectiveness of our algorithm in removing large occluding objects, as well as thin scratches. Robustness with respect to the shape of the manually selected target region is also demonstrated. Our results compare favorably to those obtained by existing techniques.

An algorithm for the simultaneous filling-in of texture and structure in regions of missing image information is presented in this paper. The basic idea is to first decompose the image into the sum of two functions with different basic characteristics, and then reconstruct each one of these functions separately with structure and texture filling-in algorithms. The first function used in the decomposition is of bounded variation, representing the underlying image structure, while the second function captures the texture and possible noise. The region of missing information in the bounded variation image is reconstructed using image inpainting algorithms, while the same region in the texture image is filled-in with texture synthesis techniques. The original image is then reconstructed adding back these two sub-images. The novel contribution of this paper is then in the combination of these three previously developed components, image decomposition with inpainting and texture synthesis, which permits the simultaneous use of filling-in algorithms that are suited for different image characteristics. Examples on real images show the advantages of this proposed approach.

An algorithm for filling-in surface holes is introduced in this paper. The basic idea is to represent the surface of interest in implicit form, and fill-in the holes with a system of geometric partial differential equations derived from image inpainting algorithms. The framework and examples with synthetic and real data are presented.

Representing the image to be inpainted in an appropriate sparse representation dictionary, and combining elements from Bayesian statistics and modern harmonic analysis, we introduce an expectation maximization (EM) algorithm for image inpainting and interpolation. From a statistical point of view, the inpainting/interpolation can be viewed as an estimation problem with missing data. Toward this goal, we propose the idea of using the EM mechanism in a Bayesian framework, where a sparsity promoting prior penalty is imposed on the reconstructed coefficients. The EM framework gives a principled way to establish formally the idea that missing samples can be recovered/ interpolated based on sparse representations. We first introduce an easy and efficient sparserepresentation-based iterative algorithm for image inpainting. Additionally, we derive its theoretical convergence properties. Compared to its competitors, this algorithm allows a high degree of flexibility to recover different structural components in the image (piecewise smooth, curvilinear, texture, etc.). We also suggest some guidelines to automatically tune the regularization parameter.

We consider the problem of filling in missing or damaged wavelet coe#cients due to lossy image transmission or communication. The task is closely related to classical inpainting problems, but also remarkably di#ers in that the inpainting regions are in the wavelet domain. New challenges include that the resulting inpainting regions in the pixel domain are usually not well defined, as well as that degradation is often spatially inhomogeneous. Two novel variational models are proposed to meet such challenges, which combine the total variation (TV) minimization technique with wavelet representations. The associated Euler-Lagrange equations lead to nonlinear partial di#erential equations (PDE's) in the wavelet domain, and proper numerical algorithms and schemes are designed to handle their computation. The proposed models can have e#ective and automatic control over geometric features of the inpainted images, including the sharpness and curvature information of edges.

A fast scheme for wavelet-domain interpolation of lost image blocks in wireless image transmission is presented in this paper. The algorithm is designed to be compatible with the wavelet-based JPEG2000 image compression standard. In the transmission of block-coded images, fading in wireless channels and congestion in packet-switched networks can cause entire blocks to be lost. Instead of using common retransmission query protocols, we reconstruct the lost block in the wavelet-domain using the correlation between the lost block and its neighbors. The algorithm first uses a simple method to determine the presence or absence of edges in the lost block. This is followed by an interpolation scheme, designed to minimize the blockiness effect, while preserving the edges or texture in the interior of the block. The interpolation scheme minimizes the square of the error between the border coefficients of the lost block and those of its neighbors, at each transform scale. The performance of the algorithm on standard test images, its low computational overhead at the decoder, and its performance vis-a-vis other reconstruction schemes, is discussed. I.

In this paper, image compression utilizing visual redundancy is investigated. Inspired by recent advancements in image inpainting techniques, we propose an image compression framework towards visual quality rather than pixel-wise fidelity. In this framework, an original image is analyzed at the encoder side so that portions of the image are intentionally and automatically skipped. Instead, some information is extracted from these skipped regions and delivered to the decoder as assistant information in the compressed fashion. The delivered assistant information plays a key role in the proposed framework because it guides image inpainting to accurately restore these regions at the decoder side. Moreover, to fully take advantage of the assistant information, a compression-oriented edge-based inpainting algorithm is proposed for image restoration, integrating pixel-wise structure propagation and patch-wise texture synthesis. We also construct a practical system to verify the effectiveness of the compression approach in which edge map serves as assistant information and the edge extraction and region removal approaches are developed accordingly. Evaluations have been made in comparison with baseline JPEG and standard MPEG-4 AVC/H.264 intra-picture coding. Experimental results show that our system achieves up to 44 % and 33 % bits-savings, respectively, at similar visual quality levels. Our proposed framework is a promising exploration towards future image and video compression.

Abstract — This paper carves out a way to image compression that is motivated by the recent advancement in image inpainting. An image coding approach is proposed in which a number of regions of the input image are skipped at the encoder and are recovered through the inpainting process at the decoder. Furthermore, a structure-aware inpainting (SAI) method is developed to restore the skipped structural regions by taking advantage of the available portion of the decoded image. A binary structure map is extracted and compressed into the generated bit-stream to indicate the skipped regions with salient structures. By making use of the decoded texture information together with the structure map, the SAI method can recover the skipped structural regions as well as the non-structural ones effectively at the decoder. Compared with JPEG, our proposed image compression scheme allows smaller file, with the potential of up to 50 % bit-saving capability, at similar visual quality levels. I.

In this paper, we present a technique for automatic color image inpainting, the art of modifying an image-region in a non-detectable form. The main algorithm is based on the theory of projections onto convex sets (POCS). The image and its wavelet transform are projected onto each other after applying suitable constraints in each domain. This technique exploits the frequency-spatial representation provided by wavelets and utilizes the correlation between the damaged area in the image and its neighborhood. The resulting restored area is homogeneous with its surrounding and preserves the aesthetics of the image. The same technique is used for simple video restoration problems. Video frames are stacked and treated as a 3-D volume, making a natural use of inter-frame correlation. 1.

We introduce and discuss shape based models for finding the best interpolation data when reconstructing missing regions in images by means of solving the Laplace equation. The shape analysis is done in the framework of Γ-convergence, from two different points of view. First, we propose a continuous PDE model and get pointwise information on the ”importance ” of each pixel by a topological asymptotic method. Second, we introduce a finite dimensional setting into the continuous model based on fat pixels (balls with positive radius), and study by Γ-convergence the asymptotics when the radius vanishes. In this way, we obtain relevant information about the optimal distribution of the best interpolation pixels. We show that the resulting optimal data sets are identical to sets that can also be motivated using level set ideas and approximation theoretic considerations. Numerical computations are presented that confirm the usefulness of our theoretical findings for PDE-based image compression.