Abstract

The Expectation-Maximization algorithm given by Dempster et al (1977) has enjoyed considerable popularity for solving MAP estimation problems. This note derives EM from the lower bounding viewpoint (Luttrell, 1994), which better illustrates the convergence properties of the algorithm and its variants. The algorithm is illustrated with two examples: pooling data from multiple noisy sources and fitting a mixture density. 1 Introduction The Expectation-Maximization (EM) algorithm is an iterative optimization technique specifically designed for probabilistic models. It uses a different strategy than gradient descent or Newton's method and sometimes provides faster convergence. However, it is still a local technique, and so is just as susceptible to local minima. The difference between EM and gradient descent is illustrated in figure 1. Starting from the current guess, gradient descent makes a linear approximation to the objective function, then takes some step uphill. Unfortunately,...

Citations

6232	Maximum likelihood from incomplete data via the EM algorithm - Dempster, Laird, et al. - 1977
3909	Neural Networks for Pattern Recognition - Bishop - 1995
612	A view of the EM algorithm that justifies incremental, sparse, and other variants - Neal, Hinton - 1998
425	Mixture densities, maximum likelihood, and the EM algorithm - Redner, Walker - 1984
41	Nonlinear programming - Bazaraa, Sherali, et al. - 1993
9	Partitioned mixture distributions: An adaptive bayesian network for low-level image processing - Luttrell - 1994
3	Computation with the exponential family and graphical models. unpublished handouts - Buntine - 1996