DMCA
Non-asymptotic Equipartition Properties for Independent and Identically Distributed Sources,” submitted to IEEE Trans. Inform. Theory for publications (2011)
| Citations: | 6 - 2 self |
BibTeX
@MISC{Yang11non-asymptoticequipartition,
author = {En-hui Yang and Jin Meng},
title = {Non-asymptotic Equipartition Properties for Independent and Identically Distributed Sources,” submitted to IEEE Trans. Inform. Theory for publications},
year = {2011}
}
Share
 |
 |
 |
 |
||
OpenURL
Abstract
Abstract—Given an independent and identically distributed source X = {Xi}∞i=1 with finite Shannon entropy or differen-tial entropy (as the case may be) H(X), the non-asymptotic equipartition property (NEP) with respect to H(X) is estab-lished, which characterizes, for any finite block length n, how close − 1 n ln p(X1X2 · · ·Xn) is to H(X) by determining the information spectrum of X1X2 · · ·Xn, i.e., the distribution of − 1 n ln p(X1X2 · · ·Xn). Non-asymptotic equipartition properties (with respect to conditional entropy, mutual information, and relative entropy) in a similar nature can also be established [3]. These non-asymptotic equipartition properties are instrumental to the development of non-asymptotic coding (including both source and channel coding) results in information theory in the same way as the asymptotic equipartition property to all asymptotic coding theorems established so far in information theory. As an example, the NEP with respect to H(X) is used to establish a non-asymptotic fixed rate source coding theorem, which reveals, for any finite block length n, a complete picture about the tradeoff between the minimum rate of fixed rate coding of X1 · · ·Xn and error probability when the error probability is a constant, or goes to 0 with block length n at a sub-polynomial n−α, 0 < α < 1, polynomial n−α, α ≥ 1, or sub-exponential e−n α, 0 < α < 1, speed. In particular, it is shown that for any finite block length n, the minimum rate (in nats per symbol) of fixed rate coding of X1X2 · · ·Xn with error probability Θ n−α√ lnn
Keyphrases
non-asymptotic equipartition property x1x2 xn finite block length identically distributed source ieee trans error probability information theory minimum rate rate coding asymptotic equipartition property non-asymptotic fixed rate source source xi conditional entropy differen-tial entropy channel coding x1 xn information spectrum non-asymptotic coding abstract given asymptotic coding theorem complete picture mutual information block length relative entropy similar nature finite shannon entropy
