Bucklew's high rate vector quantizer mismatch result is extended from fixed-rate coding to variable rate coding using a Lagrangian formulation. It is shown that if an asymptotically (high rate) optimal sequence of variable rate codes is designed for a k-dimensional probability density function (pdf) g and then applied to another pdf f for which f/g is bounded, then the resulting mismatch or loss of performance from the optimal possible is given by the relative entropy or Kullback-Leibler divergence ||g).

Contents 1 Introduction 1 2 The fixed-rate quantization problem 2 3 Consistency of empirical design 9 4 Finite sample upper bounds 16 5 Minimax lower bounds 23 6 Fundamentals of variable-rate quantization 33 7 The Lagrangian formulation 37 8 Consistency of Lagrangian empirical design 41 9 Finite sample bounds in Lagrangian design 46 0 1 Introduction The principal goal of data compression (also known as source coding) is to replace data by a compact representation in such a manner that from this representation the original data can be reconstructed either perfectly, or with high enough accuracy. Generally, the representation is given in the form of a sequence of binary digits (bits) that can be used for efficient digital transmission or storage. Certain types of data, such as general purpose data files on a computer, require perfect reconstruction. In this case the compression procedure is called lossless, and the goal is to find a rep

We extend high-rate quantization theory to Wyner-Ziv coding, i.e., lossy source coding with side information at the decoder. Ideal Slepian-Wolf coders are assumed, thus rates are conditional entropies of quantization indices given the side information. This theory is applied to the analysis of orthonormal block transforms for Wyner-Ziv coding. A formula for the optimal rate allocation and an approximation to the optimal transform are derived. The case of noisy high-rate quantization and transform coding is included in our study, in which a noisy observation of source data is available at the encoder, but we are interested in estimating the unseen data at the decoder, with the help of side information. We implement a transform-domain Wyner-Ziv video coder that encodes frames independently but decodes them conditionally. Experimental results show that using the discrete cosine transform results in a rate-distortion improvement with respect to the pixel-domain coder. Transform coders of noisy images for different communication constraints are compared. Experimental results show that the noisy Wyner-Ziv transform coder achieves a performance close to the case in which the side information is also available at the encoder. Keywords: high-rate quantization, transform coding, side information, Wyner-Ziv coding, distributed source coding, noisy source coding 1.

An entropy-constrained quantizer Q is optimal if it minimizes the expected distortion D(Q) subject to a constraint on the output entropy H(Q). In this paper we use the Lagrangian formulation to show the existence and study the structure of optimal entropy-constrained quantizers that achieve a point on the lower convex hull of the R}. In general, an optimal entropy-constrained quantizer may have a countably infinite number of codewords.

Gauss mixtures are a popular class of models in statistics and statistical signal processing because Gauss mixtures can provide good fits to smooth densities, because they have a rich theory, because they can yield good results in applications such as classification and image segmentation, and because the can be well estimated by existing algorithms such as the EM algorithm. We here use high rate quantization theory to develop a variation of an information theoretic extremal property for Gaussian sources and its extension to Gauss mixtures. This extends a method originally used for LPC speech vector quantization to provide a clustering approach to the design of Gauss mixture models. The theory provides formulas relating minimum discrimination information (MDI) selection of Gaussian components of a Gauss mixture and the mean squared error resulting when the MDI criterion is used in an optimized robust classified vector quantizer. It also provides motivation for the use of Gauss mixture models for robust compression systems for random vectors with estimated second order moments but unknown distributions.

The Lagrangian formulation of variable-rate vector quantization is known to yield useful necessary conditions for quantizer optimality and generalized Lloyd algorithms for quantizer design. In this paper the Lagrangian formulation is demonstrated to provide a convenient framework for analyzing the empirical design of variable-rate vector quantizers. In particular, the consistency of empirical design based on minimizing the Lagrangian performance over a stationary and ergodic training sequence is shown for sources with finite second moment. The finite sample performance is also studied for independent training data and sources with bounded support. Index Terms: Variable-rate quantization, empirical design, Lagrangian performance, consistency, convergence rates.

An entropy-constrained quantizer Q is optimal if it minimizes the expected distortion D(Q) subject to a constraint on the output entropy H(Q). In this paper we use the Lagrangian formulation to show the existence and study the structure of optimal entropy-constrained quantizers that achieve a point on the lower convex hull of the operational distortion-rate function D h (R) = inf Q fD(Q) : H(Q) Rg. In general, an optimal entropy-constrained quantizer may have a countably infinite number of codewords.

We propose modifications of the Viterbi Training (VT) algorithm to estimate emission parameters in Hidden Markov Models (HMM) which are widely used in speech recognition, natural language modeling, image analysis, and bioinformatics. Our goal is to alleviate the inconsistency of VT while controlling the amount of extra computations. Specifically, we modify VT to enable it asymptotically to fix the true values of the parameters as does the EM algorithm. Our approach relies on infinite Viterbi alignment and an associated with it limiting probability distribution. We focus on mixture models, an important special case of HMM, wherein the limiting distribution can be computed exactly and be used in the adjusted VT algorithm. A simulation experiment shows that our central algorithm (VA1) can dramatically improve accuracy without much cost in computation time. We also propose VA2, a more mathematically advanced correction to VT, verify its fast convergence and high accuracy, and intend to elaborate on its computationally feasible implementations in future work.

To estimate the emission parameters in hidden Markov models one commonly uses the EM algorithm or its variation. Our primary motivation, however, is the Philips speech recognition system wherein the EM algorithm is replaced by the Viterbi training algorithm. Viterbi training is faster and computationally less involved than EM, but it is also biased and need not even be consistent. We propose an alternative to the Viterbi training – adjusted Viterbi training – that has the same order of computational complexity as Viterbi training but gives more accurate estimators. Elsewhere, we studied the adjusted Viterbi training for a special case of mixtures, supporting the theory by simulations. This paper proves the adjusted Viterbi training to be also possible for more general hidden Markov models.