The problem of error control and concealment in video communication is becoming increasingly important because of the growing interest in video delivery over unreliable channels such as wireless networks and the Internet. This paper reviews the techniques that have been developed for error control and concealment in the past ten to fifteen years. These techniques are described in three categories according to the roles that the encoder and decoder play in the underlying approaches. Forward error concealment includes methods that add redundancy at the source end to enhance error resilience of the coded bit streams. Error concealment by postprocessing refers to operations at the decoder to recover the damaged areas based on characteristics of image and video signals. Finally, interactive error concealment covers techniques that are dependent on a dialog between the source and destination. Both current research activities and practice in international standards are covered.

We study the robust estimation of missing regions in images and video using adaptive, sparse reconstructions. Our primary application is on missing regions of pixels containing textures, edges, and other image features that are not readily handled by prevalent estimation and recovery algorithms. We assume that we are given a linear transform that is expected to provide sparse decompositions over missing regions such that a portion of the transform coe#cients over missing regions are zero or close to zero. We adaptively determine these small magnitude coe#cients through thresholding, establish sparsity constraints, and estimate missing regions in images using information surrounding these regions. Unlike prevalent algorithms, our approach does not necessitate any complex preconditioning, segmentation, or edge detection steps, and it can be written as a sequence of denoising operations. We show that the region types we can e#ectively estimate in a mean squared error sense are those for which the given transform provides a close approximation using sparse nonlinear approximants. We show the nature of the constructed estimators and how these estimators relate to the utilized transform and its sparsity over regions of interest. The developed estimation framework is general, and can readily be applied to nonstationary signals with a suitable choice of linear transforms. Part I discusses fundamental issues, and Part II is devoted to adaptive algorithms with extensive simulation examples that demonstrate the power of the proposed techniques.

Real-time audio and video data streamed over unreliable IP networks, such as the Internet, may encounter losses due to dropped packets or late arrivals. This paper reviews error-concealment schemes developed for streaming realtime audio and video data over the Internet. Based on their interactions with (video or audio) source coders, we classify existing techniques into source coder-independent schemes that treat underlying source coders as black boxes, and source coder-dependent schemes that exploit coder-specific characteristics to perform reconstruction. Last, we identify possible future research directions. 1. Introduction Increases in bandwidth and computational speed lead to growing interests in real-time audio and video transmissions over the Internet. In the Internet, packets carrying real-time data may be dropped or arrive too late to be useful because the Internet is a packet-switched, best-effort delivery service, with no guarantee on the quality of service (QoS). Traditi...

While there is an increasing demand for streaming video applications on the Internet, various network characteristics make the deployment of these applications more challenging than traditional Internet applications like email and the Web. The applications that transmit data over the Internet must cope with the time-varying bandwidth and delay characteristics of the Internet and must be resilient to packet loss. This thesis examines these challenges and presents a system design and implementation that ameliorates some of the important problems with video streaming over the Internet. Video

In this paper, reconstruction of missing data in transmission of baseline JPEG coded images in error prone environments is investigated. A method for estimating the missing DC coecients of blocks in a JPEG coded image which is required for decoding the compressed image, is suggested and evaluated. As effects of errors in estimating the missing DC values will appear as a number of stripes across the image, a technique for removing such stripes is also developed. Finally, the missing data is reconstructed by exploiting the correlation between adjacent blocks. A novel reconstruction technique which achieves good performance in reconstruction of edges is proposed. Simulation results indicate that our algorithms perform very well. The key contributions of the above reconstruction method are that, unlike previously published methods, it does not assume non-differential encoding of the DC values of the DCT coecients, and it performs well in the reconstruction of diagonal edges.

A new class of image information-restoration algorithms virtually different from traditional techniques are proposed. In comparison with other approaches, our methods not only use the information in local areas, but also that in the remote regions in the image. The methods originate from the idea that there exists abundant long-range correlation within natural images and the human vision systems composed of our eyes and brains can sufficiently utilize such types of information redundancy to implement the functions of image interpretation, representation, restoration, enhancement, and error concealment. Our general approach can be summarized as five basic steps: fetching, searching, matching, competing, and recovering. The experimental results on several practical applications show that our methods perform substantially better than many other state-of-the-art methods.

Resynchronizing variable-length codes (RVLCs) for large alphabets are designed by first creating resynchronizing Huffman codes and then adding an extended synchronizing codeword, and the RVLCs are applied to both JPEG and wavelet-based image compression. The RVLCs demonstrate the desired resynchronization properties, both at a symbol level and structurally so that decoded data can be correctly placed within an image following errors. The encoded images, when subject to both structural and statistical error detection and concealment, can tolerate BERs of up to and are very tolerant of burst errors. The RVLC-JPEG images have negligible overhead at visually lossless bit rates, while the RVLC-wavelet overhead can be adjusted based on the desired tolerance to burst errors and typically ranges from 7-18%. The tolerance to both bit and burst errors demonstrates that images coded with such RVLCs can be transmitted over imperfect channels suffering bit errors or packet losses without channel co...

A fast scheme for wavelet-domain interpolation of lost image blocks in wireless image transmission is presented in this paper. The algorithm is designed to be compatible with the wavelet-based JPEG2000 image compression standard. In the transmission of block-coded images, fading in wireless channels and congestion in packet-switched networks can cause entire blocks to be lost. Instead of using common retransmission query protocols, we reconstruct the lost block in the wavelet-domain using the correlation between the lost block and its neighbors. The algorithm first uses a simple method to determine the presence or absence of edges in the lost block. This is followed by an interpolation scheme, designed to minimize the blockiness effect, while preserving the edges or texture in the interior of the block. The interpolation scheme minimizes the square of the error between the border coefficients of the lost block and those of its neighbors, at each transform scale. The performance of the algorithm on standard test images, its low computational overhead at the decoder, and its performance vis-a-vis other reconstruction schemes, is discussed. I.

Transmission of digital subband coded images over lossy packet networks presents a reconstruction problem at the decoder. This paper presents two techniques for reconstruction of lost subband coefficients, one for low frequency coefficients and one for high frequency coefficients. The low frequency reconstruction algorithm is based on inherent properties of the hierarchical subband decomposition. To maintain smoothness and exploit the high intraband correlation, a cubic interpolative surface is fit to known coefficients to interpolate lost coefficients. Accurate edge placement, crucial for visual quality, is achieved by adapting the interpolation grid in both the horizontal and vertical directions as determined by the edges present. An edge model is used to characterize the adaptation, and a quantitative analysis of this model demonstrates that edges can be identified by simply examining the high frequency bands, without requiring any additional processing of the low frequency band. Hi...

Packetized image communication over lossy channels presents a reconstruction problem at the decoder. To date, reconstruction algorithms have been developed for fixed coding techniques. This paper examines the dual problem --- a block-based coding technique, namely a family of lapped orthogonal transforms (LOTs), is designed to maximize the reconstruction performance of a fixed reconstruction algorithm. Mean-reconstruction, in which missing coefficient blocks are replaced with averages of their available neighbors, is selected for simplicity and implementation ease. A reconstruction criterion is defined in terms of mean-squared error, and is used to define the reconstruction gain. A family of LOTs is then designed to consider both reconstruction gain and coding gain. Reconstruction capability and compression are exchanged, and the LOT family consists of transforms that provide increasing reconstruction capability with lower coding gain. A transform can therefore be selected based on ch...