Results 1  10
of
31
Constructing Free Energy Approximations and Generalized Belief Propagation Algorithms
 IEEE Transactions on Information Theory
, 2005
"... Important inference problems in statistical physics, computer vision, errorcorrecting coding theory, and artificial intelligence can all be reformulated as the computation of marginal probabilities on factor graphs. The belief propagation (BP) algorithm is an efficient way to solve these problems t ..."
Abstract

Cited by 414 (12 self)
 Add to MetaCart
Important inference problems in statistical physics, computer vision, errorcorrecting coding theory, and artificial intelligence can all be reformulated as the computation of marginal probabilities on factor graphs. The belief propagation (BP) algorithm is an efficient way to solve these problems that is exact when the factor graph is a tree, but only approximate when the factor graph has cycles. We show that BP fixed points correspond to the stationary points of the Bethe approximation of the free energy for a factor graph. We explain how to obtain regionbased free energy approximations that improve the Bethe approximation, and corresponding generalized belief propagation (GBP) algorithms. We emphasize the conditions a free energy approximation must satisfy in order to be a “valid ” or “maxentnormal ” approximation. We describe the relationship between four different methods that can be used to generate valid approximations: the “Bethe method, ” the “junction graph method, ” the “cluster variation method, ” and the “region graph method.” Finally, we explain how to tell whether a regionbased approximation, and its corresponding GBP algorithm, is likely to be accurate, and describe empirical results showing that GBP can significantly outperform BP.
Collective classification in network data
, 2008
"... Numerous realworld applications produce networked data such as web data (hypertext documents connected via hyperlinks) and communication networks (people connected via communication links). A recent focus in machine learning research has been to extend traditional machine learning classification te ..."
Abstract

Cited by 98 (27 self)
 Add to MetaCart
Numerous realworld applications produce networked data such as web data (hypertext documents connected via hyperlinks) and communication networks (people connected via communication links). A recent focus in machine learning research has been to extend traditional machine learning classification techniques to classify nodes in such data. In this report, we attempt to provide a brief introduction to this area of research and how it has progressed during the past decade. We introduce four of the most widely used inference algorithms for classifying networked data and empirically compare them on both synthetic and realworld data. 1
Free energy estimates of allatom protein structures using generalized belief propagation
 IN PROCEEDINGS OF THE 11TH ANNUAL INTERNATIONAL CONFERENCE ON RESEARCH IN COMPUTATIONAL MOLECULAR BIOLOGY
, 2007
"... We present a technique for approximating the free energy of protein structures using Generalized Belief Propagation (GBP). The accuracy and utility of these estimates are then demonstrated in two different application domains. First, we show that the entropy component of our free energy estimates ca ..."
Abstract

Cited by 26 (12 self)
 Add to MetaCart
We present a technique for approximating the free energy of protein structures using Generalized Belief Propagation (GBP). The accuracy and utility of these estimates are then demonstrated in two different application domains. First, we show that the entropy component of our free energy estimates can be used to distinguish native protein structures from decoys — structures with similar internal energy to that of the native structure, but otherwise incorrect. Our method is able to correctly identify the native fold from among a set of decoys with 87.5 % accuracy over a total of 48 different immunoglobin folds. The remaining 12.5 % of native structures are ranked among the top 4 of all structures. Second, we show that our estimates of ∆∆G upon mutation for three different data sets have linear correlations between 0.640.69 with experimental values and statistically significant pvalues. Together, these results suggests that GBP is an effective means for computing free energy in allatom models of protein structures. GBP is also efficient, taking a few minutes to run on a typical sized protein, further suggesting that GBP may be an attractive alternative to more costly molecular dynamic simulations for some tasks.
Convexity Arguments for Efficient Minimization of the Bethe and Kikuchi Free Energies
"... Loopy and generalized belief propagation are popular algorithms for approximate inference in Markov random fields and Bayesian networks. Fixed points of these algorithms have been shown to correspond to extrema of the Bethe and Kikuchi free energy, both of which are approximations of the exact Helmh ..."
Abstract

Cited by 26 (0 self)
 Add to MetaCart
Loopy and generalized belief propagation are popular algorithms for approximate inference in Markov random fields and Bayesian networks. Fixed points of these algorithms have been shown to correspond to extrema of the Bethe and Kikuchi free energy, both of which are approximations of the exact Helmholtz free energy. However, belief propagation does not always converge, which motivates approaches that explicitly minimize the Kikuchi/Bethe free energy, such as CCCP and UPS. Here we describe a class of algorithms that solves this typically nonconvex constrained minimization problem through a sequence of convex constrained minimizations of upper bounds on the Kikuchi free energy. Intuitively one would expect tighter bounds to lead to faster algorithms, which is indeed convincingly demonstrated in our simulations. Several ideas are applied to obtain tight convex bounds that yield dramatic speedups over CCCP.
Belief Propagation On Partially Ordered Sets
 Mathematical Systems Theory in Biology, Communications, Computation, and Finance
, 2002
"... In this paper, which is based on the important recent work of Yedidia, Freeman, and Weiss, we present a generalized form of belief propagation, viz. belie) e propagation on a partially ordered set (PBP). PBP is an iterative messagepassing algorithm for solving, either exactly or approximately, the ..."
Abstract

Cited by 25 (0 self)
 Add to MetaCart
In this paper, which is based on the important recent work of Yedidia, Freeman, and Weiss, we present a generalized form of belief propagation, viz. belie) e propagation on a partially ordered set (PBP). PBP is an iterative messagepassing algorithm for solving, either exactly or approximately, the marginalized product density problem, which is a general computational problem of wide applicability. We will show that PBP can be thought of as an algorithm for minimizing a certain "free energy" function, and by exploiting this interpretation, we will exhibit a onetoone correspondence between the fixed points of PBP and the stationary points of the free energy.
Belief Propagation and Statistical Physics
 Princeton University
, 2002
"... It was shown recently in [1] that there is a close connection between the belief propagation algorithm and certain approximations to the variational free energy in statistical physics. Specifically, the fixed points of the belief propagation algorithm are shown to coincide with the stationary points ..."
Abstract

Cited by 23 (2 self)
 Add to MetaCart
It was shown recently in [1] that there is a close connection between the belief propagation algorithm and certain approximations to the variational free energy in statistical physics. Specifically, the fixed points of the belief propagation algorithm are shown to coincide with the stationary points of the Bethe's approximate free energy subject to consistency constraints. Bethe's approximation is known as a special case of a general class of approximations called Kikuchi free energy approximations. A general class of belief propagation algorithms was also introduced in [1], which attempts to find the stationary points of a general Kikuchi free energy functional.
An edge deletion semantics for belief propagation and its practical impact on approximation quality
 In AAAI
, 2006
"... We show in this paper that the influential algorithm of iterative belief propagation can be understood in terms of exact inference on a polytree, which results from deleting enough edges from the original network. We show that deleting edges implies adding new parameters into a network, and that the ..."
Abstract

Cited by 16 (9 self)
 Add to MetaCart
We show in this paper that the influential algorithm of iterative belief propagation can be understood in terms of exact inference on a polytree, which results from deleting enough edges from the original network. We show that deleting edges implies adding new parameters into a network, and that the iterations of belief propagation are searching for values of these new parameters which satisfy intuitive conditions that we characterize. The new semantics lead to the following question: Can one improve the quality of approximations computed by belief propagation by recovering some of the deleted edges, while keeping the network easy enough for exact inference? We show in this paper that the answer is yes, leading to another question: How do we choose which edges to recover? To answer, we propose a specific method based on mutual information which is motivated by the edge deletion semantics. Empirically, we provide experimental results showing that the quality of approximations can be improved without incurring much additional computational cost. We also show that recovering certain edges with low mutual information may not be worthwhile as they increase the computational complexity, without necessarily improving the quality of approximations.
A variational approach for approximating bayesian networks by edge deletion
 In Proceedings of the Twenty Second Conference on Uncertainty in Artificial Intelligence (UAI’06
"... We consider in this paper the formulation of approximate inference in Bayesian networks as a problem of exact inference on an approximate network that results from deleting edges (to reduce treewidth). We have shown in earlier work that deleting edges calls for introducing auxiliary network paramete ..."
Abstract

Cited by 13 (4 self)
 Add to MetaCart
We consider in this paper the formulation of approximate inference in Bayesian networks as a problem of exact inference on an approximate network that results from deleting edges (to reduce treewidth). We have shown in earlier work that deleting edges calls for introducing auxiliary network parameters to compensate for lost dependencies, and proposed intuitive conditions for determining these parameters. We have also shown that our earlier method corresponds to Iterative Belief Propagation (IBP) when enough edges are deleted to yield a polytree, and corresponds to some generalizations of IBP when fewer edges are deleted. In this paper, we propose a different criteria for determining auxiliary parameters based on optimizing the KL– divergence between the original and approximate networks. We discuss the relationship between the two methods for selecting parameters, shedding new light on IBP and its generalizations. We also discuss the application of our new method to approximating inference problems which are exponential in constrained treewidth, including MAP and nonmyopic value of information. 1
Estimation and marginalization using Kikuchi approximation methods
 Neural Computation
, 2003
"... In this paper we examine a general method of approximation, – known as Kikuchi approximation method, – for finding the marginals of a productdistribution. The Kikuchi approximation method defines a certain constrained optimization problem, called the Kikuchi problem, and treats its stationary po ..."
Abstract

Cited by 12 (2 self)
 Add to MetaCart
In this paper we examine a general method of approximation, – known as Kikuchi approximation method, – for finding the marginals of a productdistribution. The Kikuchi approximation method defines a certain constrained optimization problem, called the Kikuchi problem, and treats its stationary points as approximations to the desired marginals. In this paper we show how to associate a graph to any Kikuchi problem, and describe a class of local messagepassing algorithms along the edges of any such graph, which attempt to find the solutions to the problem. We give conditions under which such algorithms converge to a stationary point of the optimization problem. Implementation of these algorithms on graphs with fewer edges require fewer operations in each iteration. We therefore characterize minimal graphs for a Kikuchi problem, which are those with the minimum number of edges, and show that all such minimal graphs have the same number of loops and share several important connectivity properties. We show that if the minimal graph is cyclefree, then Kikuchi approximation method is exact and the converse is also true generically; together with the fact that in the cyclefree case the abovementioned iterative algorithms are equivalent to the wellknown belief propagation algorithm, our results imply that, generically, Kikuchi approximation method can be exact if and only if traditional junction tree methods could also solve the problem exactly.