Abstract

Abstract. We calculate the Shannon entropy rate of a binary Hidden Markov Process (HMP), of given transition rate and noise ɛ (emission), as a series expansion in ɛ. The first two orders are calculated exactly. We then evaluate, for finite histories, simple upper-bounds of Cover and Thomas. Surprisingly, we find that for a fixed order k and history of n steps, the bounds become independent of n for large enough n. This observation is the basis of a conjecture, that the upper-bound obtained for n ≥ (k + 3)/2 gives the exact entropy rate for any desired order k of ɛ. 1 Introduction and Statement of Results Let X = {Xn}n≥1 be a first order stationary Markov process over a binary alphabet, with a symmetric transition matrix P ≡ Pab given by P00 = P11 = p = 1 − P01 = 1 − P10, where Pab = Pr(Xn = b|Xn−1 = a), ∀a, b ∈ {0, 1}. Consider also a Bernoulli

Citations