The history of the theory and practice of quantization dates to 1948, although similar ideas had appeared in the literature as long ago as 1898. The fundamental role of quantization in modulation and analog-to-digital conversion was first recognized during the early development of pulsecode modulation systems, especially in the 1948 paper of Oliver, Pierce, and Shannon. Also in 1948, Bennett published the first high-resolution analysis of quantization and an exact analysis of quantization noise for Gaussian processes, and Shannon published the beginnings of rate distortion theory, which would provide a theory for quantization as analog-to-digital conversion and as data compression. Beginning with these three papers of fifty years ago, we trace the history of quantization from its origins through this decade, and we survey the fundamentals of the theory and many of the popular and promising techniques for quantization.

this paper. Let us place it within the neural network perspective, and particularly that of learning. The area of neural networks has greatly benefited from its unique position at the crossroads of several diverse scientific and engineering disciplines including statistics and probability theory, physics, biology, control and signal processing, information theory, complexity theory, and psychology (see [45]). Neural networks have provided a fertile soil for the infusion (and occasionally confusion) of ideas, as well as a meeting ground for comparing viewpoints, sharing tools, and renovating approaches. It is within the ill-defined boundaries of the field of neural networks that researchers in traditionally distant fields have come to the realization that they have been attacking fundamentally similar optimization problems.

A fundamental problem in the transmission of analog information across a noisy discrete channel is the choice of channel code rate that optimally allocates the available transmission rate between lossy source coding and block channel coding. We establish tight bounds on the channel code rate that minimizes the average distortion of a vector quantizer cascaded with a channel coder and a binary-symmetric channel. Analytic expressions are derived in two cases of interest: small bit-error probability and arbitrary source vector dimension; arbitrary bit-error probability and large source vector dimension. We demonstrate that the optimal channel code rate is often substantially smaller than the channel capacity, and obtain a noisy-channel version of the Zador high-resolution distortion formula. Index Terms---Combined source and channel coding, error exponents, high-resolution vector quantization, separation theorem. I. INTRODUCTION S UPPOSE a lossy source coder (vector quantizer) takes an...

We explore joint source-channel coding (JSCC) for time-varying channels using a multiresolution framework for both source coding and transmission via novel multiresolution modulation constellations. We consider the problem of still image transmission over time-varying channels with the channel state information (CSI) available at 1) receiver only and 2) both transmitter and receiver being informed about the state of the channel, and we quantify the effect of CSI availability on the performance. Our source model is based on the wavelet image decomposition, which generates a collection of subbands modeled by the family of generalized Gaussian distributions. We describe an algorithm that jointly optimizes the design of the multiresolution source codebook, the multiresolution constellation, and the decoding strategy of optimally matching the source resolution and signal constellation resolution “trees” in

Asymptotically optimal zero-delay vector quantization in the presence of channel noise is studied using random coding techniques. First, an upper bound is derived for the average r th - power distortion of channel optimized k-dimensional vector quantization at transmission rate R on a binary symmetric channel with bit error probability ffl. The upper bound asymptotically equals 2 \GammarRg(ffl;k;r) , where k k+r h 1 \Gamma log 2 i 1 + 2 p ffl(1 \Gamma ffl) ji g(ffl; k; r) 1 for all ffl 0, lim ffl!0 g(ffl; k; r) = 1, and lim k!1 g(ffl; k; r) = 1. Numerical computations of g(ffl; k; r) are also given. This result is analogous to Zador's asymptotic distortion rate of 2 \GammarR for quantization on noiseless channels. Next, using a random coding argument on nonredundant index assignments, a useful upper bound is derived in terms of point density functions, on the minimum mean squared error of high resolution, regular, vector quantizers in the presence of channel noise. T...

The efficient representation and encoding of signals with limited resources, e.g., finite storage capacity and restricted transmission bandwidth, is a fundamental problem in technical as well as biological information processing systems. Typically, under realistic circumstances, the encoding and communication of messages has to deal with different sources of noise and disturbances. In this paper, we propose a unifying approach to data compression by robust vector quantization, which explicitly deals with channel noise, bandwidth limitations, and random elimination of prototypes. The resulting algorithm is able to limit the detrimental effect of noise in a very general communication scenario. In addition, the presented model allows us to derive a novel competitive neural networks algorithm, which covers topology preserving feature maps, the so-called neural-gas algorithm, and the maximum entropy soft-max rule as special cases. Furthermore, continuation methods based on these noise models impr...

We determine analytic expressions for the performance of some low-complexity combined source-channel coding systems. The main tool used is the Hadamard transform. In particular, we obtain formulas for the average distortion of binary lattice vector quantization with affine index assignments, linear block channel coding, and a binary-symmetric channel. The distortion formulas are specialized to nonredundant channel codes for a binary-symmetric channel, and then extended to affine index assignments on a binary-asymmetric channel. Various structured index assignments are compared. Our analytic formulas provide a computationally efficient method for determining the performance of various coding schemes. One interesting result shown is that for a uniform source and uniform quantizer, the Natural Binary Code is never optimal for a nonsymmetric channel, even though it is known to be optimal for a symmetric channel. Index Terms--- Index assignment, lattices, linear error-correcting codes, sou...

We propose three new design algorithms for jointly optimizing source and channel codes. Our optimality criterion is to minimize the average end-to-end distortion. For a given channel SNR and transmission rate, our joint source and channel code designs achieve an optimal allocation of bits between the source and channel coders. Our three techniques include a sourceoptimized channel code, a channel-optimized source code, and an iterative descent technique combining the design strategies of the other two codes. The joint designs use channel-optimized vector quantization (COVQ) for the source code and rate-compatible punctured convolutional (RCPC) coding for the channel code. The optimal bit allocation reduces distortion by up to 6 dB over suboptimal allocations and by up to 4 dB relative to standard COVQ for the source data set considered. We find that all three code designs have roughly the same performance when their bit allocations are optimized. This result follows from the fact that at the optimal bit allocation the channel code removes most of the channel errors, in which case the three design techniques are roughly equivalent. We also compare the robustness of the three techniques to channel mismatch. We conclude the paper by relaxing the fixed transmission rate constraint and jointly optimizing the transmission rate, source code, and channel code. Index Terms---Joint source/channel coding, optimal bit allocation, RCPC channel code, vector quantization. I.

this document.

We propose algorithms to design channel-optimized vector quantizers in the presence of channel mismatch. We consider two cases: (i) no information about the statistics of the channel bit error rate is available and (ii) the probability density function of the channel bit error rate is known. We also consider the use of an estimate of the channel signalto -noise ratio to improve performance. Simulation results demonstrate the advantages of new design algorithms. 1. INTRODUCTION Source coding applications which involve transmission over noisy channels have been the main motivation for studying the sensitivity of a vector quantizer (VQ) to channel noise. These studies have led to the development of techniques for making a VQ robust with respect to channel noise, either by an appropriate binary codeword assignment [1, 2] or by a complete redesign of the VQ partition and codebook, resulting in the so-called channel-optimized vector quantizer (COVQ) [3, 4]. Previous studies on the subject h...