Results 1  10
of
27
Logconcavity and the maximum entropy property of the Poisson distribution
 Stochastic Processes and their Applications
"... We prove that the Poisson distribution maximises entropy in the class of ultralogconcave distributions, extending a result of Harremoës. The proof uses ideas concerning logconcavity, and a semigroup action involving adding Poisson variables and thinning. We go on to show that the entropy is a conc ..."
Abstract

Cited by 22 (10 self)
 Add to MetaCart
We prove that the Poisson distribution maximises entropy in the class of ultralogconcave distributions, extending a result of Harremoës. The proof uses ideas concerning logconcavity, and a semigroup action involving adding Poisson variables and thinning. We go on to show that the entropy is a concave function along this semigroup. 1 Maximum entropy distributions It is wellknown that the distributions which maximise entropy under certain very natural conditions take a simple form. For example, among random variables with fixed mean and variance the entropy is maximised by the normal distribution. Similarly, for random variables with positive support and fixed mean, the entropy is maximised by the exponential distribution. The standard technique for proving such results uses the Gibbs inequality, and exploits the fact that, given a function f(x), and fixing Λ(p) = ∫ p(x)f(x)dx, the maximum entropy density is of the form α exp(−βf(x)). Example 1.1 For a density p with mean µ and variance σ2, write φµ,σ2 for the density of a N(µ, σ2) random variable, and define the function Λ(p) = − ∫ p(x) log φ µ,σ2(x)dx.
Information theoretic proofs of entropy power inequalities,” arXiv:0704.1751v1 [cs.IT
, 2007
"... Abstract—While most useful information theoretic inequalities can be deduced from the basic properties of entropy or mutual information, up to now Shannon’s entropy power inequality (EPI) is an exception: Existing information theoretic proofs of the EPI hinge on representations of differential entro ..."
Abstract

Cited by 14 (2 self)
 Add to MetaCart
Abstract—While most useful information theoretic inequalities can be deduced from the basic properties of entropy or mutual information, up to now Shannon’s entropy power inequality (EPI) is an exception: Existing information theoretic proofs of the EPI hinge on representations of differential entropy using either Fisher information or minimum meansquare error (MMSE), which are derived from de Bruijn’s identity. In this paper, we first present an unified view of these proofs, showing that they share two essential ingredients: 1) a data processing argument applied to a covariancepreserving linear transformation; 2) an integration over a path of a continuous Gaussian perturbation. Using these ingredients, we develop a new and brief proof of the EPI through a mutual information inequality, which replaces Stam and Blachman’s Fisher information inequality (FII) and an inequality for MMSE by Guo, Shamai, and Verdú used in earlier proofs. The result has the advantage of being very simple in that it relies only on the basic properties of mutual information. These ideas are then generalized to various extended versions of the EPI: Zamir and Feder’s generalized EPI for linear transformations of the random variables, Takano and Johnson’s EPI for dependent variables, Liu and Viswanath’s covarianceconstrained EPI, and Costa’s concavity inequality for the entropy power. Index Terms—Data processing inequality, de Bruijn’s identity, differential entropy, divergence, entropy power inequality (EPI),
Fisher information, compound Poisson approximation and the Poisson channel
 in Proc. IEEE International Symposium on Information Theory
, 2007
"... Abstract — Fisher information plays a fundamental role in the analysis of Gaussian noise channels and in the study of Gaussian approximations in probability and statistics. For discrete random variables, the scaled Fisher information plays an analogous role in the context of Poisson approximation. O ..."
Abstract

Cited by 13 (2 self)
 Add to MetaCart
Abstract — Fisher information plays a fundamental role in the analysis of Gaussian noise channels and in the study of Gaussian approximations in probability and statistics. For discrete random variables, the scaled Fisher information plays an analogous role in the context of Poisson approximation. Our first results show that it also admits a minimum mean squared error characterization with respect to the Poisson channel, and that it satisfies a monotonicity property that parallels the monotonicity recently established for the central limit theorem in terms of Fisher information. We next turn to the more general case of compound Poisson distributions on the nonnegative integers, and we introduce two new “local information quantities ” to play the role of Fisher information in this context. We show that they satisfy subadditivity properties similar to those of classical Fisher information, we derive a minimum mean squared error characterization, and we explore their utility for obtaining compound Poisson approximation bounds. I.
1 On the Entropy of Compound Distributions on Nonnegative Integers
"... Abstract—Some entropy comparison results are presented concerning compound distributions on nonnegative integers. The main result shows that, under a logconcavity assumption, two compound distributions are ordered in terms of Shannon entropy if both the “numbers of claims ” and the “claim sizes ” a ..."
Abstract

Cited by 6 (5 self)
 Add to MetaCart
Abstract—Some entropy comparison results are presented concerning compound distributions on nonnegative integers. The main result shows that, under a logconcavity assumption, two compound distributions are ordered in terms of Shannon entropy if both the “numbers of claims ” and the “claim sizes ” are ordered accordingly in the convex order. Several maximum/minimum entropy theorems follow as a consequence. Most importantly, two recent results of Johnson et al. (2008) on maximum entropy characterizations of compound Poisson and compound binomial distributions are proved under fewer assumptions and with simpler arguments. Index Terms—compound binomial, compound Poisson, convex order, infinite divisibility, logconcavity, maximum entropy, minimum
Thinning and information projections
 IEEE Symp. Inf. Theory
, 2008
"... Abstract — In the law of thin numbers we are interested in lower bounds on the information divergence from a thinned distribution to a Poisson distribution. Information projections turns out to be the right tool to use. Conditions for the existence of information projections to exist are given. A me ..."
Abstract

Cited by 5 (1 self)
 Add to MetaCart
Abstract — In the law of thin numbers we are interested in lower bounds on the information divergence from a thinned distribution to a Poisson distribution. Information projections turns out to be the right tool to use. Conditions for the existence of information projections to exist are given. A method of translating results related to Poisson distributions into results related to Gaussian distributions is developed and used to prove a new nontrivial result related to the central limit theorem. I.
Monotonic convergence in an informationtheoretic law of small numbers,” Preprint, 2009, http://arxiv.org/abs/0810.5203 Yaming Yu received the B.S. degree in mathematics from
 Beijing University
, 2005
"... Abstract—A version of the law of small numbers is analyzed in informationtheoretic terms. Specifically, let f = {fi, i = 0, 1,...} be a probability mass function (pmf) on nonnegative integers with mean λ < ∞. Denote the nth convolution of f by f ∗n and denote the αthinning of f by Tα(f). Then, as ..."
Abstract

Cited by 5 (4 self)
 Add to MetaCart
Abstract—A version of the law of small numbers is analyzed in informationtheoretic terms. Specifically, let f = {fi, i = 0, 1,...} be a probability mass function (pmf) on nonnegative integers with mean λ < ∞. Denote the nth convolution of f by f ∗n and denote the αthinning of f by Tα(f). Then, as n → ∞, the entropy H(T1/n(f ∗n)) tends to H(po(λ)), where po(λ) denotes the pmf of the Poisson distribution with mean λ, and the relative entropy D(T1/n(f ∗n)po(λ)) tends to zero, if it ever becomes finite. Moreover, α −1 D(Tα(f)po(αλ)) increases in α ∈ (0,1), and n −1 D (f ∗n po(nλ)) decreases in n = 1, 2,.... It follows that D(T1/n(f ∗n)po(λ)) decreases monotonically in n. Furthermore, assuming that f is ultralogconcave, we show that H(T1/n(f ∗n)) increases monotonically in n. This is a discrete analogue of the monotonicity of entropy considered by Artstein et al. (2004). A convergence rate is also obtained. If f is either ultralogconcave or has finite support, then D(T1/n(f ∗n)po(λ)) = O(n −2), as n → ∞. In general, our results extend the parallel between the informationtheoretic central limit theorem and the informationtheoretic law of small numbers explored by Kontoyiannis et al. (2005) and Harremoës et al. (2007, 2008). Ingredients in the proofs include convexity, majorization, and stochastic orders. Index Terms—binomial distribution; convex order; logarithmic Sobolev inequality; majorization; Poisson approximation; relative entropy; Schurconcavity; stochastic orders; thinning; ultralogconcavity. I.
On the entropy and logconcavity of compound Poisson measures
"... Motivated, in part, by the desire to develop an informationtheoretic foundation for compound Poisson approximation limit theorems (analogous to the corresponding developments for the central limit theorem and for simple Poisson approximation), this work examines sufficient conditions under which th ..."
Abstract

Cited by 4 (1 self)
 Add to MetaCart
Motivated, in part, by the desire to develop an informationtheoretic foundation for compound Poisson approximation limit theorems (analogous to the corresponding developments for the central limit theorem and for simple Poisson approximation), this work examines sufficient conditions under which the compound Poisson distribution has maximal entropy within a natural class of probability measures on the nonnegative integers. We show that the natural analog of the Poisson maximum entropy property remains valid if the measures under consideration are logconcave, but that it fails in general. A parallel maximum entropy result is established for the family of compound binomial measures. The proofs are largely based on ideas related to the semigroup approach introduced in recent work by Johnson [12] for the Poisson family. Sufficient conditions are given for compound distributions to be logconcave, and specific examples are presented illustrating all the above results.
Compound Poisson approximation via information functionals
 Electron. J. Probab
, 2010
"... An informationtheoretic development is given for the problem of compound Poisson approximation, which parallels earlier treatments for Gaussian and Poisson approximation. Let PSn be the distribution of a sum Sn = ∑n i=1 Yi of independent integervalued random variables Yi. Nonasymptotic bounds are ..."
Abstract

Cited by 4 (0 self)
 Add to MetaCart
An informationtheoretic development is given for the problem of compound Poisson approximation, which parallels earlier treatments for Gaussian and Poisson approximation. Let PSn be the distribution of a sum Sn = ∑n i=1 Yi of independent integervalued random variables Yi. Nonasymptotic bounds are derived for the distance between PSn and an appropriately chosen compound Poisson law. In the case where all Yi have the same conditional distribution given {Yi = 0}, a bound on the relative entropy distance between PSn and the compound Poisson distribution is derived, based on the dataprocessing property of relative entropy and earlier Poisson approximation results. When the Yi have arbitrary distributions, corresponding bounds are derived in terms of the total variation distance. The main technical ingredient is the introduction of two “information functionals, ” and the analysis of their properties. These information functionals play a role analogous to that of the classical Fisher information in normal approximation. Detailed comparisons are made between the resulting inequalities and related bounds. AMS classification — 60E15, 60E07, 60F05, 94A17
Monotonicity, thinning and discrete versions of the Entropy Power Inequality
, 2009
"... We consider the entropy of sums of independent discrete random variables, in analogy with Shannon’s Entropy Power Inequality, where equality holds for normals. In our case, infinite divisibility suggests that equality should hold for Poisson variables. We show that some natural analogues of the Entr ..."
Abstract

Cited by 3 (0 self)
 Add to MetaCart
We consider the entropy of sums of independent discrete random variables, in analogy with Shannon’s Entropy Power Inequality, where equality holds for normals. In our case, infinite divisibility suggests that equality should hold for Poisson variables. We show that some natural analogues of the Entropy Power Inequality do not in fact hold, but propose an alternative formulation which does always hold. The key to many proofs of Shannon’s Entropy Power Inequality is the behaviour of entropy on scaling of continuous random variables. We believe that Rényi’s operation of thinning discrete random variables plays a similar role to scaling, and give a sharp bound on how the entropy of ultra logconcave random variables behaves on thinning. In the spirit of the monotonicity results established by Artstein, Ball, Barthe and Naor, we prove a stronger version of concavity of entropy, which implies a strengthened form of our discrete Entropy Power Inequality.
Concavity of entropy under thinning
 In Proceedings of ISIT 2009, 28th June  3rd July 2009, Seoul
, 2009
"... Abstract—Building on the recent work of Johnson (2007) and Yu (2008), we prove that entropy is a concave function with respect to the thinning operation Tα. That is, if X and Y are independent random variables on Z+ with ultralogconcave probability mass functions, then H(TαX + T1−αY) ≥ αH(X) + (1 ..."
Abstract

Cited by 3 (1 self)
 Add to MetaCart
Abstract—Building on the recent work of Johnson (2007) and Yu (2008), we prove that entropy is a concave function with respect to the thinning operation Tα. That is, if X and Y are independent random variables on Z+ with ultralogconcave probability mass functions, then H(TαX + T1−αY) ≥ αH(X) + (1 − α)H(Y), 0 ≤ α ≤ 1, where H denotes the discrete entropy. This is a discrete analogue of the inequality (h denotes the differential entropy) h ( √ αX + √ 1 − αY) ≥ αh(X) + (1 − α)h(Y), 0 ≤ α ≤ 1, which holds for continuous X and Y with finite variances and is equivalent to Shannon’s entropy power inequality. As a consequence we establish a special case of a conjecture of Shepp and Olkin (1981). Possible extensions are also discussed. Index Terms—binomial thinning; convolution; entropy power inequality; Poisson distribution; ultralogconcavity. I.