Results 1  10
of
25
Good ErrorCorrecting Codes based on Very Sparse Matrices
, 1999
"... We study two families of errorcorrecting codes defined in terms of very sparse matrices. "MN" (MacKayNeal) codes are recently invented, and "Gallager codes" were first investigated in 1962, but appear to have been largely forgotten, in spite of their excellent properties. The decoding of both cod ..."
Abstract

Cited by 513 (25 self)
 Add to MetaCart
We study two families of errorcorrecting codes defined in terms of very sparse matrices. "MN" (MacKayNeal) codes are recently invented, and "Gallager codes" were first investigated in 1962, but appear to have been largely forgotten, in spite of their excellent properties. The decoding of both codes can be tackled with a practical sumproduct algorithm. We prove that these codes are "very good," in that sequences of codes exist which, when optimally decoded, achieve information rates up to the Shannon limit. This result holds not only for the binarysymmetric channel but also for any channel with symmetric stationary ergodic noise. We give experimental results for binarysymmetric channels and Gaussian channels demonstrating that practical performance substantially better than that of standard convolutional and concatenated codes can be achieved; indeed, the performance of Gallager codes is almost as close to the Shannon limit as that of turbo codes.
Message passing algorithms for compressed sensing: I. motivation and construction
 Proc. ITW
, 2010
"... Abstract—In a recent paper, the authors proposed a new class of lowcomplexity iterative thresholding algorithms for reconstructing sparse signals from a small set of linear measurements [1]. The new algorithms are broadly referred to as AMP, for approximate message passing. This is the second of tw ..."
Abstract

Cited by 67 (9 self)
 Add to MetaCart
Abstract—In a recent paper, the authors proposed a new class of lowcomplexity iterative thresholding algorithms for reconstructing sparse signals from a small set of linear measurements [1]. The new algorithms are broadly referred to as AMP, for approximate message passing. This is the second of two conference papers describing the derivation of these algorithms, connection with related literature, extensions of original framework, and new empirical evidence. This paper describes the state evolution formalism for analyzing these algorithms, and some of the conclusions that can be drawn from this formalism. We carried out extensive numerical simulations to confirm these predictions. We present here a few representative results. I. GENERAL AMP AND STATE EVOLUTION We consider the model
Efficient learning in Boltzmann Machines using linear response theory
 Neural Computation
, 1997
"... The learning process in Boltzmann Machines is computationally very expensive. The computational complexity of the exact algorithm is exponential in the number of neurons. We present a new approximate learning algorithm for Boltzmann Machines, which is based on mean field theory and the linear respon ..."
Abstract

Cited by 44 (5 self)
 Add to MetaCart
The learning process in Boltzmann Machines is computationally very expensive. The computational complexity of the exact algorithm is exponential in the number of neurons. We present a new approximate learning algorithm for Boltzmann Machines, which is based on mean field theory and the linear response theorem. The computational complexity of the algorithm is cubic in the number of neurons. In the absence of hidden units, we show how the weights can be directly computed from the fixed point equation of the learning rules. Thus, in this case we do not need to use a gradient descent procedure for the learning process. We show that the solutions of this method are close to the optimal solutions and give a significant improvement when correlations play a significant role. Finally, we apply the method to a pattern completion task and show good performance for networks up to 100 neurons. 1 Introduction Boltzmann Machines (BMs) (Ackley et al., 1985), are networks of binary neurons with a stoc...
Combinatorial Landscapes
 SIAM REVIEW
, 2002
"... Fitness landscapes have proven to be a valuable concept in evolutionary biology, combinatorial optimization, and the physics of disordered systems. A fitness landscape is a mapping from a configuration space into the real numbers. The configuration space is equipped with some notion of adjacency, ne ..."
Abstract

Cited by 33 (2 self)
 Add to MetaCart
Fitness landscapes have proven to be a valuable concept in evolutionary biology, combinatorial optimization, and the physics of disordered systems. A fitness landscape is a mapping from a configuration space into the real numbers. The configuration space is equipped with some notion of adjacency, nearness, distance or accessibility. Landscape theory has emerged as an attempt to devise suitable mathematical structures for describing the "static" properties of landscapes as well as their influence on the dynamics of adaptation. In this review we focus on the connections of landscape theory with algebraic combinatorics and random graph theory, where exact results are available.
A mean field spin glass with short range interactions
 Comm. Math. Phys
, 1986
"... Abstract. We formulate and study a spin glass model on the Bethe lattice. Appropriate boundary fields replace the traditional selfconsistent methods; they give our model welldefined thermodynamic properties. We establish that there is a spin glass transition temperature above which the singlesite ..."
Abstract

Cited by 23 (3 self)
 Add to MetaCart
Abstract. We formulate and study a spin glass model on the Bethe lattice. Appropriate boundary fields replace the traditional selfconsistent methods; they give our model welldefined thermodynamic properties. We establish that there is a spin glass transition temperature above which the singlesite magnetizations vanish, and below which the EdwardsAnderson order parameter is strictly positive. In a neighborhood below the transition temperature, we use bifurcation theory to establish the existence of a nontrivial distribution of singlesite magnetizations. Two properties of this distribution are studied: the leading perturbative correction to the Gaussian scaling form at the transition, and the (nonperturbative) behavior of the tails.
Bound Propagation
 Journal of Artificial Intelligence Research
, 2003
"... In this article we present an algorithm to compute bounds on the marginals of a graphical model. For several small clusters of nodes upper and lower bounds on the marginal values are computed independently of the rest of the network. The range of allowed probability distributions over the surroundin ..."
Abstract

Cited by 20 (0 self)
 Add to MetaCart
In this article we present an algorithm to compute bounds on the marginals of a graphical model. For several small clusters of nodes upper and lower bounds on the marginal values are computed independently of the rest of the network. The range of allowed probability distributions over the surrounding nodes is restricted using earlier computed bounds. As we will show, this can be...
Mean Field Theory for Graphical Models
, 2000
"... Introduction During the last few years, the use of probabilistic methods in artificial intelligence and machine learning has gained enormous popularity. In particular, probabilistic graphical models have become the preferred method for knowledge representation and reasoning [1]. The advantage of th ..."
Abstract

Cited by 11 (2 self)
 Add to MetaCart
Introduction During the last few years, the use of probabilistic methods in artificial intelligence and machine learning has gained enormous popularity. In particular, probabilistic graphical models have become the preferred method for knowledge representation and reasoning [1]. The advantage of the probabilistic approach is that all assumptions are made explicit in the modeling process and that consequences, such as predictions on novel data, are assumption free and follow from a mechanistic computation. The drawback of the probabilistic approach is that the method is intractable. This means that the typical computation scales exponentially with the problem size. Recently, a number of authors have proposed methods for approximate inference in large graphical models. The simplest approach gives a lower bound on the probability of a subset of variables using Jenssen's inequality [2]. The method involves the minimization of the KL divergence between the target probability distri
Analysis of Bit Error Probability of DirectSequence CDMA Multiuser Demodulators
 in Advances in Neural Information Processing Systems
, 2001
"... We analyze the bit error probability of multiuser demodulators for directsequence binary phaseshiftkeying (DS/BPSK) CDMA channel with additive gaussian noise. The problem of multiuser demodulation is cast into the finitetemperature decoding problem, and replica analysis is applied to evaluate th ..."
Abstract

Cited by 10 (1 self)
 Add to MetaCart
We analyze the bit error probability of multiuser demodulators for directsequence binary phaseshiftkeying (DS/BPSK) CDMA channel with additive gaussian noise. The problem of multiuser demodulation is cast into the finitetemperature decoding problem, and replica analysis is applied to evaluate the performance of the resulting MPM (Marginal Posterior Mode) demodulators, which include the optimal demodulator and the MAP demodulator as special cases. An approximate implementation of demodulators is proposed using analogvalued Hopfield model as a naive meanfield approximation to the MPM demodulators, and its performance is also evaluated by the replica analysis. Results of the performance evaluation shows effectiveness of the optimal demodulator and the meanfield demodulator compared with the conventional one, especially in the cases of small information bit rate and low noise level.
A deterministic annealing framework for unsupervised texture segmentation
 Tech. Rep. IAITR962
, 1996
"... We present a novel framework for unsupervised texture segmentation, which relies on statistical tests as a measure of homogeneity. Texture segmentation is formulated as a pairwise data clustering problem with a sparse neighborhood structure. The pairwise dissimilarities of texture blocks are compute ..."
Abstract

Cited by 7 (1 self)
 Add to MetaCart
We present a novel framework for unsupervised texture segmentation, which relies on statistical tests as a measure of homogeneity. Texture segmentation is formulated as a pairwise data clustering problem with a sparse neighborhood structure. The pairwise dissimilarities of texture blocks are computed using a multiscale image representation based on Gabor filters, which are tuned to spatial frequencies at different scales and orientations. We derive and discuss a family of objective functions to pose the segmentation problem in a precise mathematical formulation. An efficient optimization method, known as deterministic annealing, is applied to solve the associated optimization problem. The general framework of deterministic annealing and meanfield approximation is introduced and the canonical way to derive efficient algorithms within this framework is described in detail. Moreover the combinatorial optimization problem is examined from the viewpoint of scale space theory. The novel algorithm has been extensively tested on Brodatzlike microtexture mixtures and on realword images. In addition, benchmark studies with alternative segmentation techniques are reported.
InformationGeometrical Significance of Sparsity in Gallager Codes
 Advances in Neural Information Processing Systems
, 2001
"... We report a result of perturbation analysis on decoding error of the belief propagation decoder for Gallager codes. The analysis is based on information geometry, and it shows that the principal term of decoding error at equilibrium comes from the membedding curvature of the loglinear submanif ..."
Abstract

Cited by 6 (6 self)
 Add to MetaCart
We report a result of perturbation analysis on decoding error of the belief propagation decoder for Gallager codes. The analysis is based on information geometry, and it shows that the principal term of decoding error at equilibrium comes from the membedding curvature of the loglinear submanifold spanned by the estimated pseudoposteriors, one for the full marginal, and K for partial posteriors, each of which takes a single check into account, where K is the number of checks in the Gallager code. It is then shown that the principal error term vanishes when the paritycheck matrix of the code is so sparse that there are no two columns with overlap greater than 1.