this article. Some further techniques which go somewhat beyond this model will also be discussed. [Begin Sidebar Inset Article #1] A History of Existing Visual Coding Standards

Abstract—Resilience to packet loss is a critical requirement in predictive video coding for transmission over packet-switched networks, since the prediction loop propagates errors and causes substantial degradation in video quality. This work proposes an algorithm to optimally estimate the overall distortion of decoder frame reconstruction due to quantization, error propagation, and error concealment. The method recursively computes the total decoder distortion at pixel level precision to accurately account for spatial and temporal error propagation. The accuracy of the estimate is demonstrated via simulation results. The estimate is integrated into a rate-distortion (RD)-based framework for optimal switching between intra-coding and inter-coding modes per macroblock. The cost in computational complexity is modest. The framework is further extended to optimally exploit feedback/acknowledgment information from the receiver/network. Simulation results both with and without a feedback channel demonstrate that precise distortion estimation enables the coder to achieve substantial and consistent gains in PSNR over known state-of-the-art RD- and non-RDbased mode switching methods. Index Terms—Intra-mode, intra-refresh, packet loss, packet video, rate-distortion, video coding.

Content delivery over the Internet needs to address both the multimedia nature of the content and the capabilities of the diverse client platforms the content is being delivered to. We present a system that adapts multimedia Web documents to optimally match the capabilities of the client device requesting it. This system has two key components: (1) A representation scheme called the InfoPyramid that provides a multi-modal, multi-resolution representation hierarchy for multimedia. (2) A customizer that selects the best content representation to meet the client capabilities while delivering the most value. We model the selection process as a resource allocation problem in a generalized rate-distortion framework. In this framework, we address the issue of both multiple media types in a Web document and multiple resource types at the client. We extend this framework to allow prioritization on the content items in a Web document. We illustrate our content adaptation technique with a web ...

Long-term memory prediction extends the spatial displacement vector utilized in hybrid video coding by a variable time delay permitting the use of more than one reference frame for motion compensation. This extension provides improved rate-distortion performance. However, motion compensation in combination with transmission errors leads to temporal error propagation that occurs when the reference frames at encoder and decoder dier. In this paper, we present a framework that incorporates an error estimate into rate-constrained motion estimation and mode decision. Experimental results with a Rayleigh fading channel show that long-term memory motion compensation signicantly outperforms single-frame prediction. 1. INTRODUCTION The eciency of long-term memory motion-compensated prediction (MCP) as an approach to improve coding performance has been demonstrated in [1]. The ITU-T has decided to adopt this feature as Annex U to version 3 of the H.263 standard. In this paper, we show that t...

Video transmission in wireless environments is a challenging task calling for high-compression efficiency as well as a network friendly design. Both have been major goals of the H.264/AVC standardization effort addressing "conversational" (i.e., video telephony) and "nonconversational" (i.e., storage, broadcast, or streaming) applications. The video compression performance of the H.264/AVC video coding layer typically provides a significant improvement. The network-friendly design goal of H.264/AVC is addressed via the network abstraction layer that has been developed to transport the coded video data over any existing and future networks including wireless systems. The main objective of this paper is to provide an overview over the tools which are likely to be used in wireless environments and discusses the most challenging application, wireless conversational services in greater detail. Appropriate justifications for the application of different tools based on experimental results are presented.

Digital video's increased popularity has been driven to a large extent by a flurry of recently proposed international standards (MPE(-i, MPE(-2, H.263, etc.). In most standards, the rate control scheme, which plays an important role in improving and stabilizing the decoding and play-back quality, is not defined and thus different strategies can be implemented in each encoder design. Several rate-distortion (R-D) based techniques have been proposed to aim at the best possible quality for a given channel rate and buffer size. These approaches are complex because they require the R-D characteristics of the input data to be measured before making quantization assignment decisions. In this paper, we show how the complexity of computing the R-D data can be reduced without significantly reducing the performance of the optimization procedure. We propose two methods which provide successive reductions in com- plexity by (i) using models to interpolate the rate and distortion characteristics, and (ii) using past frames instead of current ones to determine the models. Our first method is applicable to situations (e.g. broadcast video) where a long encoding delay is possible, while our second approach is more useful for computation-constrained interactive video applications. The first method can also be used to benchmark other approaches. Both methods can achieve over peak signal-to-noise rate (PSNR) gain over simple methods like the MPE( Test Model 5 (TMS) rate control, with even greater gains during scene change transitions. In addition, both methods make few a priori assumptions and provide robustness in their performance over a range of video sources and encoding rates. In terms of complexity, our first algorithm roughly doubles the encoding time as compared to simpler techniqu...

Long-term memory prediction extends the spatial displacement vector utilized in hybrid video coding by a variable time delay permitting the use of more than one reference frame for motion compensation. This extension leads to improved rate-distortion performance. However, motion compensation in combination with transmission errors leads to temporal error propagation that occurs when the reference frames at coder and decoder differ. In this paper, we present a framework that incorporates an estimated error into rate-constrained motion estimation and mode decision. Experimental results with a Rayleigh fading channel show that long-term memory prediction significantly outperforms the single-frame prediction H.263-based anchor. When a feedback channel is available, the decoder can inform the encoder about successful or unsuccessful transmission events by sending positive (ACK) or negative (NACK) acknowledgments. This information is utilized for updating the error estimates at the encoder. Similar concepts such as the ACK and NACK mode known from the H.263 standard are unified into a general framework providing superior transmission performance.

This paper describes an algorithm which can achieve a constant bit rate when coding multiple video objects. The implementation is a nontrivial extension of the MPEG-4 rate control algorithm for single video objects which employs a quadratic ratequantizer model. The algorithm is organized into two stages: a pre- and a postencoding stage. In the preencoding stage, an initial target estimate is made for each object. Based on the buffer fullness, the total target is adjusted and then distributed proportional to the relative size, motion, and variance of each object. Based on the new individual targets and rate-quantizer relation for texture, appropriate quantization parameters are calculated. After each object is encoded, the model parameters for each object are updated, and if necessary, frames are skipped to ensure that the buffer does not overflow. A preframeskip control is exercised to avoid buffer overflow when the motion and shape information occupies a significant portion of the bit budget. The rate control algorithm switches between two operation modes so that the coder can reduce the spatial coding accuracy for an improved temporal resolution. A shape-coding control mechanism is also proposed, which provides a tradeoff between texture and shape coding accuracy. Overall, the algorithm is able to successfully achieve the target bit rate, effectively code arbitrarily shaped objects, and maintain a stable buffer level. These techniques have been adopted by the MPEG committee in July 1997 as part of the video Verification Model (VM8). Index Terms--- Bit allocation, buffering policy, multiple video objects, rate control, shape coding control. I.

We consider the problem of error control for receiver-driven layered multicast of audio and video over the Internet. The sender injects into the network multiple source layers and multiple channel coding (parity) layers, some of which are delayed relative to the source. Each receiver subscribes to the number of source layers and the number of parity layers that optimizes the receiver's quality for its available bandwidth and packet loss probability. We augment this layered FEC system with layered pseudo-ARQ. Although feedback is normally problematic in broadcast situations, ARQ can be simulated by having the receivers subscribe and unsubscribe to the delayed parity layers to receive missing information. This pseudo-ARQ scheme avoids an implosion of repeat requests at the sender and is scalable to an unlimited number of receivers.

Over the last one and a half decades, digital video compression technologies have become an integral part of the way we create, communicate, and consume visual information. In this paper, techniques for video compression are reviewed, starting from basic concepts. The rate-distortion performance of modern video compression schemes is the result of an interaction between motion representation techniques, intra-picture prediction techniques, waveform coding of differences, and waveform coding of various refreshed regions. The paper starts with an explanation of the basic concepts of video codec design and then explains how these various features have been integrated into international standards, up to and including the most recent such standard, known as H.264/AVC.