Recently, researchers have demonstrated that "loopy belief propagation" --- the use of Pearl's polytree algorithm in a Bayesian network with loops --- can perform well in the context of error-correcting codes. The most dramatic instance of this is the near Shannon-limit performance of "Turbo Codes" --- codes whose decoding algorithm is equivalent to loopy belief propagation in a chain-structured Bayesian network. In this paper we ask: is there something special about the error-correcting code context, or does loopy propagation work as an approximate inference scheme in a more general setting? We compare the marginals computed using loopy propagation to the exact ones in four Bayesian network architectures, including two real-world networks: ALARM and QMR. We find that the loopy beliefs often converge and when they do, they give a good approximation to the correct marginals. However, on the QMR network, the loopy beliefs oscillated and had no obvious relationship ...

Abstract—In this paper, we will describe the close connection between the now celebrated iterative turbo decoding algorithm of Berrou et al. and an algorithm that has been well known in the artificial intelligence community for a decade, but which is relatively unknown to information theorists: Pearl’s belief propagation algorithm. We shall see that if Pearl’s algorithm is applied to the “belief network ” of a parallel concatenation of two or more codes, the turbo decoding algorithm immediately results. Unfortunately, however, this belief diagram has loops, and Pearl only proved that his algorithm works when there are no loops, so an explanation of the excellent experimental performance of turbo decoding is still lacking. However, we shall also show that Pearl’s algorithm can be used to routinely derive previously known iterative, but suboptimal, decoding algorithms for a number of other error-control systems, including Gallager’s

This article analyzes the behavior of local propagation rules in graphical models with a loop.

Abstract-The performance of Turbo codes is addressed by examining the code’s distance spectrum. The “error floor ” that occurs at moderate signal-to-noise ratios is shown to be a conse-quence of the relatively low free distance of the code. It is also shown that the “error floor ” can be lowered by increasing the size of the interleaver without changing the free distance of the code. Alternatively, the free distance of the code may be increased by using primitive feedback polynomials. The excellent performance of lurbo codes at low signal-to-noise ratios is explained in terms of the distance spectrum. The interleaver in the Turbo encoder is shown to reduce the number of low-weight codewords through a process called “spectral thinning. ” This thinned distance spec-trum results in the free distance asymptote being the dominant performance parameter for low and moderate signal-to-noise ratios. Index Terms-Turbo codes, convolutional codes, distance spec-trum. T I.

Local belief propagation rules of the sort proposed by Pearl (1988) are guaranteed to converge to the optimal beliefs for singly connected networks. Recently, a number of researchers have empirically demonstrated good performance of these same algorithms on networks with loops, but a theoretical understanding of this performance has yet to be achieved. Here we lay a foundation for an understanding of belief propagation in networks with loops. For networks with a single loop, we derive an analytical relationship between the steady state beliefs in the loopy network and the true posterior probability. Using this relationship we show a category of networks for which the MAP estimate obtained by belief update and by belief revision can be proven to be optimal (although the beliefs will be incorrect). We show how nodes can use local information in the messages they receive in order to correct the steady state beliefs. Furthermore we prove that for all networks with a single loop, the MAP es...

Belief propagation (BP) was only supposed to work for tree-like networks but works surprisingly well in many applications involving networks with loops, including turbo codes. However, there has been little understanding of the algorithm or the nature of the solutions it nds for general graphs. We show that BP can only converge to a stationary point of an approximate free energy, known as the Bethe free energy in statistical physics. This result characterizes BP xed-points and makes connections with variational approaches to approximate inference. More importantly, our analysis lets us build on the progress made in statistical physics since Bethe's approximation was introduced in 1935. Kikuchi and others have shown how to construct more accurate free energy approximations, of which Bethe's approximation is the simplest. Exploiting the insights from our analysis, we derive generalized belief propagation (GBP) versions of these Kikuchi approximations. These new message passing algorithms can be signicantly more accurate than ordinary BP, at an adjustable increase in complexity. We illustrate such a new GBP algorithm on a grid Markov network and show that it gives much more accurate marginal probabilities than those found using ordinary BP.

It is now understood [7, 8] that the turbo decoding algorithm is an instance of a probability propagation algorithm (PPA) on a graph with many cycles. However, PPA-type algorithms are known to give exact results only when the underlying graph is cycle-free. Thus it is important to study the "approximate correctness" of PPA on graphs with cycles. In this paper we make a first step by discussing the behavior of an PPA in graphs with a single cycle. This work is directly relevant to the study of iterative decoding of tail-biting codes, whose underlying graph has just one cycle [3], [12]. First, we shall show that for strictly positive local kernels, the iterations of the PPA will always converge to the same fixed point regardless of the scheduling order used. Moreover, the length of the cycle does not play a role in this convergence. Secondly, we shall generalize a result of McEliece and Rodemich [9], by showing that if the hidden variables in the cycle are binary-valued, a decision based...

Soft-input soft-output building blocks #modules# are presented to construct and iteratively decode in a distributed fashion code networks, a new concept that includes, and generalizes, various forms of concatenated coding schemes. Among the modules, a central role is played by the SISO module #and the underlying algorithm#: it consists of a four-port device performing a processing of the sequences of two input probability distributions by constraining them to the code trellis structure. The SISO and other soft-input soft-output modules are employed to construct and decode a variety of code networks, including "turbo codes" and serially concatenated codes with interleavers. Keywords Iterative decoding, turbo codes, serial concatenated codes, soft decoding algorithms. I. Introduction This paper concerns the construction and the distributed, iterative decoding of a conglomerate of codes that we call code networks, the name stemming from the complexity and richness of the poss...

MAP is the problem of finding a most probable instantiation of a set of variables in a Bayesian network, given some evidence. MAP appears

MAP is the problem of finding a most probable instantiation of a set of variables given evidence. MAP has always been perceived to be significantly harder than the related problems of computing the probability of a variable instantiation (Pr), or the problem of computing the most probable explanation (MPE). This paper investigates the complexity of MAP in Bayesian networks. Specifically, we show that MAP is complete for NP PP and provide further negative complexity results for algorithms based on variable elimination. We also show that MAP remains hard even when MPE and Pr become easy. For example, we show that MAP is NP-complete when the networks are restricted to polytrees, and even then can not be effectively approximated. Given the difficulty of computing MAP exactly, and the difficulty of approximating MAP while providing useful guarantees on the resulting approximation, we investigate best effort approximations. We introduce a generic MAP approximation framework. We provide two instantiations of the framework; one for networks which are amenable to exact inference (Pr), and one for networks for which even exact inference is too hard. This allows MAP approximation on networks that are too complex to even exactly solve the easier problems, Pr and MPE. Experimental results indicate that using these approximation algorithms provides much better solutions than standard techniques, and provide accurate MAP estimates in many cases. 1.