A discrete denoising algorithm estimates the input sequence to a discrete memoryless channel (DMC) based on the observation of the entire output sequence. For the case in which the DMC is known and the quality of the reconstruction is evaluated with a given single-letter fidelity criterion, we propose a discrete denoising algorithm that does not assume knowledge of statistical properties of the input sequence. Yet, the algorithm is universal in the sense of asymptotically performing as well as the optimum denoiser that knows the input sequence distribution, which is only assumed to be stationary and ergodic. Moreover, the algorithm is universal also in a semi-stochastic setting, in which the input is an individual sequence, and the randomness is due solely to the channel noise.

Abstract—The goal of a denoising algorithm is to recover a signal from its noise-corrupted observations. Perfect recovery is seldom possible and performance is measured under a given single-letter fidelity criterion. For discrete signals corrupted by a known discrete memoryless channel (DMC), the Discrete Universal DEnoiser (DUDE) was recently shown to perform this task asymptotically optimally, without knowledge of the statistical properties of the source. In the present work, we address the scenario where, in addition to the lack of knowledge of the source statistics, there is also uncertainty in the channel characteristics. We propose a family of discrete denoisers and establish their asymptotic optimality under a minimax performance criterion which we argue is appropriate for this setting. As we show elsewhere, the proposed schemes can also be implemented computationally efficiently. Index Terms—Denoising, denoising algorithms, discrete universal denoising, Discrete Universal DEnoiser (DUDE), estimation, minimax schemes. I.

Let X = (X1 , . . .) be a stationary ergodic finite-alphabet source, X denote its first n symbols, and Y be the codeword assigned to X by a lossy source code. The empirical kth-order joint distribution Q ) along the pair (X ). Our main interest is in the sample behavior of this (random) distribution. Letting I(Q when (X we show that for any (sequence of) lossy source code(s) of rate a.s., where H(X) denotes the entropy rate of X. This is shown to imply, for a large class of sources including all i.i.d. sources and all sources satisfying the Shannon lower bound with equality, that for any sequence of codes which is good in the sense of asymptotically attaining a point on the rate distortion curve P X k , Y k a.s., whenever P X k , Y k is the unique distribution attaining the minimum in the definition of the kth-order rate distortion function. Further consequences of these results are explored. These include a simple proof of Kie#er's sample converse to lossy source coding, as well as pointwise performance bounds for compression-based denoisers. 1

Consider the problem of rate-constrained reconstruction of a finite-alphabet discrete memoryless signal X (X1 , . . . , Xn ), based on a noise-corrupted observation sequence Z , which is the finite-alphabet output of a Discrete Memoryless Channel (DMC) whose input is X . Suppose that there is some uncertainty in the source distribution, in the channel characteristics, or in both. Equivalently, suppose that the distribution of the pairs (X i , Z i ), rather than completely being known, is only known to belong to a set #. Suppose further that the relevant performance criterion is the probability of excess distortion, i.e., letting ) denote the reconstruction, we are interested in the behavior of P # , where # is a (normalized) block distortion induced by a single-letter distortion measure and P # denotes the probability measure corresponding to the case where (X i , Z i ) #, # #.

Abstract—We characterize the rate distortion function for the source coding with decoder side information setting when the ith reconstruction symbol is allowed to depend only on the first i + ` side information symbols, for some finite look-ahead `, in addition to the index from the encoder. For the case of causal side information, i.e., ` = 0, we find that the penalty of causality is the omission of the subtracted mutual information term in the Wyner–Ziv rate distortion function. For `> 0, we derive a computable “infinite-letter” expression for the rate distortion function. When specialized to the near-lossless case, our results characterize the best achievable rate for the Slepian–Wolf source coding problem with finite side information looka-head, and have some surprising implications. We find that side information is useless for any fixed ` when the joint probability mass function (PMF) of the source and side information satisfies the positivity condition P (x; y)> 0 for all (x; y). More generally, the optimal rate depends on the distribution of the pair X; Y only through the distribution of X and the bipartite graph whose edges represent the pairs x; y for which P (x; y)> 0. On the other hand, if side information look-ahead is allowed to grow faster than logarithmic in the block length, then H(X j Y) is achievable. Finally, we apply our approach to derive a computable expression for channel capacity when state information is available at the encoder with limited look-ahead. Index Terms—Causal source codes, delay-constrained coding, Gel’fand–Pinsker channel, rate distortion function, Slepian–Wolf coding, Wyner–Ziv coding. I.

Abstract — The problem of statistical learning is to construct an accurate predictor of a random variable as a function of a correlated random variable on the basis of an i.i.d. training sample from their joint distribution. Allowable predictors are constrained to lie in some specified class, and the goal is to approach asymptotically the performance of the best predictor in the class. We consider two settings in which the learning agent only has access to rate-limited descriptions of the training data, and present information-theoretic bounds on the predictor performance achievable in the presence of these communication constraints. Our proofs do not assume any separation structure between compression and learning and rely on a new class of operational criteria specifically tailored to joint design of encoders and learning algorithms in rate-constrained settings. I.