## Steepest Descent as Message Passing

Citations: | 2 - 0 self |

### BibTeX

@MISC{Dauwels_steepestdescent,

author = {Justin Dauwels and Sascha Korl and Hans-andrea Loeliger},

title = {Steepest Descent as Message Passing},

year = {}

}

### OpenURL

### Abstract

Abstract — It is shown how steepest descent (or steepest ascent) may be viewed as a message passing algorithm with “local ” message update rules. For example, the well-known backpropagation algorithm for the training of feed-forward neural networks may be viewed as message passing on a factor graph. The factor graph approach with its emphasis on “local ” computations makes it easy to combine steepest descent with other message passing algorithms such as the sum/max-product algorithms, expectation maximization, Kalman filtering/smoothing, and particle filters. As an example, parameter estimation in a state space model is considered. For this example, it is shown how steepest descent can be used for the maximization step in expectation maximization.

### Citations

4815 |
Neural networks for pattern recognition
- Bishop
- 1995
(Show Context)
Citation Context ...network theory: indeed, if one applies the rule (22) together with (7) (13) and (14) to a factor graph that represents a feed-forward neural network, one obtains the popular backpropagation algorithm =-=[8]-=-. D. Summary We have seen that 1) When steepest descent is combined with the sumproduct algorithm, gradients of sum-product messages are required. � � ��� �� � �� ��� Fig. 3. Deterministic mapping �. ... |

740 |
Neuro-dynamic programming. Athena Scientific
- Bertsekas, Tsitsiklis
- 1996
(Show Context)
Citation Context ... Suppose we want to find � ��max � � � � � � � � (1) where � takes values in . The familiar steepest descent (or “steepest ascent” or “gradient descent/ascent”) method tries to find � �max as follows =-=[1]-=-: starting from some initial guess � � , compute �� � � � � � �� �� � �� � � � (2) for � � � � ����, where the parameter �� (the “step size”) is a positive real number that may be depend on �. An alte... |

121 | An introduction to factor graphs
- Loeliger
- 2004
(Show Context)
Citation Context ...) is iterated until a fixed point is reached or until the available time is over. In this paper, we will describe steepest descent as a message passing technique that operates on a factor graph. (See =-=[2]-=- for an introduction to factor graphs.) In particular, we will be interested in solving (1) in the case where � � is a “marginal” of a real-valued function � � � : � � � � � � � � (4) where denotes ei... |

55 |
Recursive parameter estimation using incomplete data
- Titterington
- 1984
(Show Context)
Citation Context ...estimate � �, but the old messages ������� � ; this type of scheduling is the main idea behind [9]. Forwardonly message passing amounts to recursive algorithms, known as “recursive EM” or “online EM” =-=[10]-=-–[13]. In [10] [11], recursive algorithms for fixed parameter estimation are derived based on EM in conjunction with steepest descent. It is common practice to extend the algorithms of [10] [11] to ti... |

22 |
Sequential Algorithms for Parameter Estimation Based on the Kullback-Leibler Information Measure
- Weinstein, Feder, et al.
- 1990
(Show Context)
Citation Context ...he old messages ������� � ; this type of scheduling is the main idea behind [9]. Forwardonly message passing amounts to recursive algorithms, known as “recursive EM” or “online EM” [10]–[13]. In [10] =-=[11]-=-, recursive algorithms for fixed parameter estimation are derived based on EM in conjunction with steepest descent. It is common practice to extend the algorithms of [10] [11] to timevarying parameter... |

21 | Rubin,“Maximum Likelihood from Incomplete Data via the EM - Dempster, Laird, et al. - 1977 |

20 |
BPhase estimation by message passing
- Dauwels, Loeliger
(Show Context)
Citation Context ...) can be evaluated in a straightforward manner. If on the other hand those variables (or a subset of them) are continuous, the integrals in (8) and (11) may be evaluated in several ways [6] (see also =-=[7]-=- for an illustration): In some cases, a closed-form expression of (8) or (11) exists. The integrals in (8) and (11) can be approximated based on canonical distributions as for example Gaussian distrib... |

14 | Some remarks on factor graphs
- Loeliger
(Show Context)
Citation Context ...monstrate the maximization step of EM by means of steepest descent as message passing. Some concluding remarks are offered in Section 4. II. SUM-PRODUCT ALGORITHM AND STEEPEST DESCENT In earlier work =-=[6]-=-, we briefly touched upon the subject of gradient descent in the context of the sum(mary)-product algorithm. Here, we present a more detailed exposition. As in [6], we start by considering the factor ... |

13 |
EM-based recursive estimation of channel parameters
- Zamiri-Jafarian, Pasupathy
- 1999
(Show Context)
Citation Context ...arying parameters by introducing some ad-hoc “forgetting” mechanism. We illustrated by the example (26)–(27), how parameters with non-trivial priors can be treated in a rigorous way (see also [4] [5] =-=[12]-=- [13]). The example (26)–(27) can easily be extended to general functions �� and ��. The gradient of the �-message leaving � the generic node � � � � � � ����� �� (cf. Fig. 7) is given by � � � � � � ... |

11 | Expectation maximization as message passing
- Dauwels, Korl, et al.
(Show Context)
Citation Context ... maximization step) is to use steepest descent. An alternative way to compute � �max (1) (exactly or approximately) is expectation maximization (EM). (A message passing view of EM is developed in [4] =-=[5]-=-.) However, the maximization step of EM is often intractable; we will show how steepest descent can be applied in such cases. This paper is structured as follows. In Section 2, we describe steepest de... |

7 | Iterative multiuser detection with graphical modeling
- Eckford, Pasupathy
- 2000
(Show Context)
Citation Context ...(the maximization step) is to use steepest descent. An alternative way to compute � �max (1) (exactly or approximately) is expectation maximization (EM). (A message passing view of EM is developed in =-=[4]-=- [5].) However, the maximization step of EM is often intractable; we will show how steepest descent can be applied in such cases. This paper is structured as follows. In Section 2, we describe steepes... |

4 |
Recursive estimate-maximize (EM) algorithms for time varying parameters with applications to multiple target tracking
- Frenkel, Feder
- 1995
(Show Context)
Citation Context ...ate � �, but the old messages ������� � ; this type of scheduling is the main idea behind [9]. Forwardonly message passing amounts to recursive algorithms, known as “recursive EM” or “online EM” [10]–=-=[13]-=-. In [10] [11], recursive algorithms for fixed parameter estimation are derived based on EM in conjunction with steepest descent. It is common practice to extend the algorithms of [10] [11] to timevar... |

3 |
Turbo synchronization: A combined sum-product and expectation-maximization algorithm approach
- Herzet, Ramon, et al.
- 2005
(Show Context)
Citation Context ... probability functions � � � �� � � � � (cf. Step 4) are then recomputed according to (30) using the new estimate � �, but the old messages ������� � ; this type of scheduling is the main idea behind =-=[9]-=-. Forwardonly message passing amounts to recursive algorithms, known as “recursive EM” or “online EM” [10]–[13]. In [10] [11], recursive algorithms for fixed parameter estimation are derived based on ... |

2 | Expectation maximization for phase estimation - Dauwels, Korl, et al. - 2005 |