Results 1  10
of
36
Loopy Belief Propagation for Approximate Inference: An Empirical Study
 In Proceedings of Uncertainty in AI
, 1999
"... Recently, researchers have demonstrated that "loopy belief propagation"  the use of Pearl's polytree algorithm in a Bayesian network with loops  can perform well in the context of errorcorrecting codes. The most dramatic instance of this is the near Shannonlimit performa ..."
Abstract

Cited by 479 (18 self)
 Add to MetaCart
Recently, researchers have demonstrated that "loopy belief propagation"  the use of Pearl's polytree algorithm in a Bayesian network with loops  can perform well in the context of errorcorrecting codes. The most dramatic instance of this is the near Shannonlimit performance of "Turbo Codes"  codes whose decoding algorithm is equivalent to loopy belief propagation in a chainstructured Bayesian network. In this paper we ask: is there something special about the errorcorrecting code context, or does loopy propagation work as an approximate inference scheme in a more general setting? We compare the marginals computed using loopy propagation to the exact ones in four Bayesian network architectures, including two realworld networks: ALARM and QMR. We find that the loopy beliefs often converge and when they do, they give a good approximation to the correct marginals. However, on the QMR network, the loopy beliefs oscillated and had no obvious relationship ...
Turbo decoding as an instance of Pearl’s belief propagation algorithm
 IEEE Journal on Selected Areas in Communications
, 1998
"... Abstract—In this paper, we will describe the close connection between the now celebrated iterative turbo decoding algorithm of Berrou et al. and an algorithm that has been well known in the artificial intelligence community for a decade, but which is relatively unknown to information theorists: Pear ..."
Abstract

Cited by 326 (15 self)
 Add to MetaCart
Abstract—In this paper, we will describe the close connection between the now celebrated iterative turbo decoding algorithm of Berrou et al. and an algorithm that has been well known in the artificial intelligence community for a decade, but which is relatively unknown to information theorists: Pearl’s belief propagation algorithm. We shall see that if Pearl’s algorithm is applied to the “belief network ” of a parallel concatenation of two or more codes, the turbo decoding algorithm immediately results. Unfortunately, however, this belief diagram has loops, and Pearl only proved that his algorithm works when there are no loops, so an explanation of the excellent experimental performance of turbo decoding is still lacking. However, we shall also show that Pearl’s algorithm can be used to routinely derive previously known iterative, but suboptimal, decoding algorithms for a number of other errorcontrol systems, including Gallager’s
Correctness of Local Probability Propagation in Graphical Models with Loops
, 2000
"... This article analyzes the behavior of local propagation rules in graphical models with a loop. ..."
Abstract

Cited by 180 (9 self)
 Add to MetaCart
This article analyzes the behavior of local propagation rules in graphical models with a loop.
A distance spectrum interpretation of turbo codes
 IEEE Trans. Inform. Theory
, 1996
"... AbstractThe performance of Turbo codes is addressed by examining the code’s distance spectrum. The “error floor ” that occurs at moderate signaltonoise ratios is shown to be a consequence of the relatively low free distance of the code. It is also shown that the “error floor ” can be lowered by ..."
Abstract

Cited by 91 (7 self)
 Add to MetaCart
AbstractThe performance of Turbo codes is addressed by examining the code’s distance spectrum. The “error floor ” that occurs at moderate signaltonoise ratios is shown to be a consequence of the relatively low free distance of the code. It is also shown that the “error floor ” can be lowered by increasing the size of the interleaver without changing the free distance of the code. Alternatively, the free distance of the code may be increased by using primitive feedback polynomials. The excellent performance of lurbo codes at low signaltonoise ratios is explained in terms of the distance spectrum. The interleaver in the Turbo encoder is shown to reduce the number of lowweight codewords through a process called “spectral thinning. ” This thinned distance spectrum results in the free distance asymptote being the dominant performance parameter for low and moderate signaltonoise ratios. Index TermsTurbo codes, convolutional codes, distance spectrum. T I.
Bethe free energy, Kikuchi approximations and belief propagation algorithms
, 2000
"... Belief propagation (BP) was only supposed to work for treelike networks but works surprisingly well in many applications involving networks with loops, including turbo codes. However, there has been little understanding of the algorithm or the nature of the solutions it nds for general graphs. ..."
Abstract

Cited by 74 (2 self)
 Add to MetaCart
Belief propagation (BP) was only supposed to work for treelike networks but works surprisingly well in many applications involving networks with loops, including turbo codes. However, there has been little understanding of the algorithm or the nature of the solutions it nds for general graphs. We show that BP can only converge to a stationary point of an approximate free energy, known as the Bethe free energy in statistical physics. This result characterizes BP xedpoints and makes connections with variational approaches to approximate inference. More importantly, our analysis lets us build on the progress made in statistical physics since Bethe's approximation was introduced in 1935. Kikuchi and others have shown how to construct more accurate free energy approximations, of which Bethe's approximation is the simplest. Exploiting the insights from our analysis, we derive generalized belief propagation (GBP) versions of these Kikuchi approximations. These new message passing algorithms can be signicantly more accurate than ordinary BP, at an adjustable increase in complexity. We illustrate such a new GBP algorithm on a grid Markov network and show that it gives much more accurate marginal probabilities than those found using ordinary BP.
Belief Propagation and Revision in Networks with Loops
, 1997
"... Local belief propagation rules of the sort proposed by Pearl (1988) are guaranteed to converge to the optimal beliefs for singly connected networks. Recently, a number of researchers have empirically demonstrated good performance of these same algorithms on networks with loops, but a theoretical und ..."
Abstract

Cited by 71 (6 self)
 Add to MetaCart
Local belief propagation rules of the sort proposed by Pearl (1988) are guaranteed to converge to the optimal beliefs for singly connected networks. Recently, a number of researchers have empirically demonstrated good performance of these same algorithms on networks with loops, but a theoretical understanding of this performance has yet to be achieved. Here we lay a foundation for an understanding of belief propagation in networks with loops. For networks with a single loop, we derive an analytical relationship between the steady state beliefs in the loopy network and the true posterior probability. Using this relationship we show a category of networks for which the MAP estimate obtained by belief update and by belief revision can be proven to be optimal (although the beliefs will be incorrect). We show how nodes can use local information in the messages they receive in order to correct the steady state beliefs. Furthermore we prove that for all networks with a single loop, the MAP es...
On the Convergence of Iterative Decoding on Graphs with a Single Cycle
 In Proc. 1998 ISIT
, 1998
"... It is now understood [7, 8] that the turbo decoding algorithm is an instance of a probability propagation algorithm (PPA) on a graph with many cycles. However, PPAtype algorithms are known to give exact results only when the underlying graph is cyclefree. Thus it is important to study the "ap ..."
Abstract

Cited by 43 (1 self)
 Add to MetaCart
It is now understood [7, 8] that the turbo decoding algorithm is an instance of a probability propagation algorithm (PPA) on a graph with many cycles. However, PPAtype algorithms are known to give exact results only when the underlying graph is cyclefree. Thus it is important to study the "approximate correctness" of PPA on graphs with cycles. In this paper we make a first step by discussing the behavior of an PPA in graphs with a single cycle. This work is directly relevant to the study of iterative decoding of tailbiting codes, whose underlying graph has just one cycle [3], [12]. First, we shall show that for strictly positive local kernels, the iterations of the PPA will always converge to the same fixed point regardless of the scheduling order used. Moreover, the length of the cycle does not play a role in this convergence. Secondly, we shall generalize a result of McEliece and Rodemich [9], by showing that if the hidden variables in the cycle are binaryvalued, a decision based...
SoftInput SoftOutput Modules for the Construction and Distributed Iterative Decoding of Code Networks
, 1998
"... Softinput softoutput building blocks #modules# are presented to construct and iteratively decode in a distributed fashion code networks, a new concept that includes, and generalizes, various forms of concatenated coding schemes. Among the modules, a central role is played by the SISO module #and t ..."
Abstract

Cited by 42 (6 self)
 Add to MetaCart
Softinput softoutput building blocks #modules# are presented to construct and iteratively decode in a distributed fashion code networks, a new concept that includes, and generalizes, various forms of concatenated coding schemes. Among the modules, a central role is played by the SISO module #and the underlying algorithm#: it consists of a fourport device performing a processing of the sequences of two input probability distributions by constraining them to the code trellis structure. The SISO and other softinput softoutput modules are employed to construct and decode a variety of code networks, including "turbo codes" and serially concatenated codes with interleavers. Keywords Iterative decoding, turbo codes, serial concatenated codes, soft decoding algorithms. I. Introduction This paper concerns the construction and the distributed, iterative decoding of a conglomerate of codes that we call code networks, the name stemming from the complexity and richness of the poss...
MAP Complexity Results and Approximation Methods
 IN PROCEEDINGS OF THE 18TH CONFERENCE ON UNCERTAINTY IN ARTIFICIAL INTELLIGENCE (UAI
, 2002
"... MAP is the problem of finding a most probable instantiation of a set of variables in a Bayesian network, given some evidence. MAP appears ..."
Abstract

Cited by 37 (2 self)
 Add to MetaCart
MAP is the problem of finding a most probable instantiation of a set of variables in a Bayesian network, given some evidence. MAP appears
Complexity results and approximation strategies for map explanations
 Journal of Artificial Intelligence Research
, 2004
"... MAP is the problem of finding a most probable instantiation of a set of variables given evidence. MAP has always been perceived to be significantly harder than the related problems of computing the probability of a variable instantiation (Pr), or the problem of computing the most probable explanatio ..."
Abstract

Cited by 33 (3 self)
 Add to MetaCart
MAP is the problem of finding a most probable instantiation of a set of variables given evidence. MAP has always been perceived to be significantly harder than the related problems of computing the probability of a variable instantiation (Pr), or the problem of computing the most probable explanation (MPE). This paper investigates the complexity of MAP in Bayesian networks. Specifically, we show that MAP is complete for NP PP and provide further negative complexity results for algorithms based on variable elimination. We also show that MAP remains hard even when MPE and Pr become easy. For example, we show that MAP is NPcomplete when the networks are restricted to polytrees, and even then can not be effectively approximated. Given the difficulty of computing MAP exactly, and the difficulty of approximating MAP while providing useful guarantees on the resulting approximation, we investigate best effort approximations. We introduce a generic MAP approximation framework. We provide two instantiations of the framework; one for networks which are amenable to exact inference (Pr), and one for networks for which even exact inference is too hard. This allows MAP approximation on networks that are too complex to even exactly solve the easier problems, Pr and MPE. Experimental results indicate that using these approximation algorithms provides much better solutions than standard techniques, and provide accurate MAP estimates in many cases. 1.