Results 1  10
of
11
Implementing and Diagnosing the Stochastic Approximation EM Algorithm
 Journal of Computational and Graphical Statistics
, 2006
"... The stochastic approximation EM (SAEM) algorithm is a simulationbased alternative to the EM (Expectation/Maximization) algorithm for situations when the Estep is hard or impossible. One of the appeals of SAEM is that, unlike other Monte Carlo versions of EM, it converges with a fixed (and typicall ..."
Abstract

Cited by 13 (2 self)
 Add to MetaCart
The stochastic approximation EM (SAEM) algorithm is a simulationbased alternative to the EM (Expectation/Maximization) algorithm for situations when the Estep is hard or impossible. One of the appeals of SAEM is that, unlike other Monte Carlo versions of EM, it converges with a fixed (and typically small) simulation size. Another appeal is that, in practice, the only decision that has to be made is the choice of the step size which is a onetime decision and which is usually done before starting the method. The downside of SAEM is that there exist no datadriven and/or modeldriven recommendations as to the magnitude of this step size. We argue in this paper that a challenging model/data combination coupled with an unlucky step size can lead to very poor algorithmic performance and, in particular, to a premature stop of the method. We propose a new heuristic for SAEM’s step size selection based on the underlying EM rate of convergence. We also use the muchappreciated EM likelihoodascent property to derive a new and flexible way of monitoring the progress of the SAEM algorithm. We apply the method to a challenging geostatistical model of online retailing.
The EM Algorithm, Its Stochastic Implementation and Global Optimization: Some Challenges and Opportunities for OR
, 2006
"... The EM algorithm is a very powerful optimization method and has reached popularity in many fields. Unfortunately, EM is only a local optimization method and can get stuck in suboptimal solutions. While more and more contemporary data/model combinations yield more than one optimum, there have been on ..."
Abstract

Cited by 10 (4 self)
 Add to MetaCart
The EM algorithm is a very powerful optimization method and has reached popularity in many fields. Unfortunately, EM is only a local optimization method and can get stuck in suboptimal solutions. While more and more contemporary data/model combinations yield more than one optimum, there have been only very few attempts at making EM suitable for global optimization. In this paper we review the basic EM algorithm, its properties and challenges and we focus in particular on its stochastic implementation. The stochastic EM implementation promises relief to some of the contemporary data/model challenges and it is particularly wellsuited for a wedding with global optimization ideas since most global optimization paradigms are also based on the principles of stochasticity. We review some of the challenges of the stochastic EM implementation and propose a new algorithm that combines the principles of EM with that of the Genetic Algorithm. While this new algorithm shows some promising results for clustering of an online auction database of functional objects, the primary goal of this work is to bridge a gap between the field of statistics, which is home to extensive research on the EM algorithm, and the field of operations research, in which work on global optimization thrives, and to stir new ideas for joint research between the two.
New Global Optimization Algorithms for ModelBased Clustering
 Journal of Computational and Graphical Statistics
"... The ExpectationMaximization (EM) algorithm is a very popular optimization tool in modelbased clustering problems. However, while the algorithm is convenient to implement and numerically very stable, it only produces solutions that are locally optimal. Thus, EM may not achieve the globally optimal ..."
Abstract

Cited by 4 (2 self)
 Add to MetaCart
(Show Context)
The ExpectationMaximization (EM) algorithm is a very popular optimization tool in modelbased clustering problems. However, while the algorithm is convenient to implement and numerically very stable, it only produces solutions that are locally optimal. Thus, EM may not achieve the globally optimal solution to clustering problems, which can have a large number of local optima. This paper introduces several new algorithms designed to produce global solutions in modelbased clustering problems. The building blocks for these algorithms are methods from the operations research literature, namely the CrossEntropy (CE) method and Model Reference Adaptive Search (MRAS). One problem with applying these two approaches directly is the efficient simulation of positive definite covariance matrices. We propose several new solutions to this problem. One solution is to apply the principles of ExpectationMaximization updating, which leads to two new algorithms, CEEM and MRASEM. We also propose two additional algorithms, CECD and MRASCD, which rely on the Cholesky decomposition. We conduct numerical experiments to evaluate the effectiveness of the proposed algorithms in comparison to classical EM. We find that although a single run of the new algorithms is slower than EM, they have the potential of producing significantly better global solutions to the modelbased clustering problem. We also show that the global optimum “matters ” in the sense that it significantly improves the clustering task. 1 1
Global Convergence of Model Reference Adaptive Search for Gaussian Mixtures. Working Paper
, 2007
"... While the ExpectationMaximization (EM) algorithm is a popular and convenient tool for mixture analysis, it only produces solutions that are locally optimal, and thus may not achieve the globally optimal solution. This paper introduces a new algorithm, based on a recently introduced global optimizat ..."
Abstract

Cited by 1 (1 self)
 Add to MetaCart
(Show Context)
While the ExpectationMaximization (EM) algorithm is a popular and convenient tool for mixture analysis, it only produces solutions that are locally optimal, and thus may not achieve the globally optimal solution. This paper introduces a new algorithm, based on a recently introduced global optimization approach called Model Reference Adaptive Search (MRAS), designed to produce globally optimal solutions of finite mixture models. We propose the MRAS mixture model algorithm for the estimation of Gaussian mixtures, which relies on the Cholesky decomposition to simulate random positive definite covariance matrices, and we provide a theoretical proof of global convergence to the optimal solution of the likelihood function. Numerical experiments illustrate the effectiveness of the proposed algorithm in finding global optima in settings where the classical EM fails to do so. 1
Dynamic Scoring of Customers using Learning Spatial Choice Models
, 2005
"... Scoring models are extensively used in CRM applications. However, while most scoring models are static in nature, the quickly changing environment in which firms operate often demands dynamic models that are able to adapt to that change. Moreover, scoring models are often used in environments that s ..."
Abstract
 Add to MetaCart
Scoring models are extensively used in CRM applications. However, while most scoring models are static in nature, the quickly changing environment in which firms operate often demands dynamic models that are able to adapt to that change. Moreover, scoring models are often used in environments that span vast and diverse geographical regions. The online channel is only one example where the underlying environment can be geographical extremely diverse. This calls for models that can take the geographical diversity into account: spatial models. In this work we propose spatial models of choice that adapt to dynamically changing environments. These learning spatial choice models incorporate new information as it becomes available and are therefore superior over static models. We estimate the model using a version of the online EM algorithm. We apply the learning spatial model of choice to an online publishing firm’s data on customer choice of print versus PDF and show how the scoring model can be useful in setting targeted ecoupon promotions or dynamic pricing application.
STRUCTURAL OPTIMIZATION USING FEMLAB AND SMOOTH SUPPORT VECTOR REGRESSION
, 2006
"... ..."
(Show Context)
Numerical integration in logisticnormal models
, 2006
"... www.elsevier.com/locate/csda Marginal maximum likelihood estimation is commonly used to estimate logisticnormal models. In this approach, the contribution of random effects to the likelihood is represented as an intractable integral over their distribution. Thus, numerical methods such as Gauss–Her ..."
Abstract
 Add to MetaCart
(Show Context)
www.elsevier.com/locate/csda Marginal maximum likelihood estimation is commonly used to estimate logisticnormal models. In this approach, the contribution of random effects to the likelihood is represented as an intractable integral over their distribution. Thus, numerical methods such as Gauss–Hermite quadrature (GH) are needed. However, as the dimensionality increases, the number of quadrature points becomes rapidly too high. A possible solution can be found among the QuasiMonte Carlo (QMC) methods, because these techniques yield quite good approximations for highdimensional integrals with a much lower number of points, chosen for their optimal location. A comparison between three integration methods for logisticnormal models: GH, QMC, and full Monte Carlo integration (MC) is presented. It turns out that, under certain conditions, the QMC and MC method perform better than the GH in terms of accuracy and computing time. © 2006 Elsevier B.V. All rights reserved.
Numerical integration in logisticnormal models
, 2005
"... When estimating logisticnormal models, the integral appearing in the marginal likelihood is analytically intractable, so that numerical methods such as GaussHermite quadrature (GH) are needed. When the dimensionality increases, the number of quadrature points becomes too high. A possible solution ..."
Abstract
 Add to MetaCart
(Show Context)
When estimating logisticnormal models, the integral appearing in the marginal likelihood is analytically intractable, so that numerical methods such as GaussHermite quadrature (GH) are needed. When the dimensionality increases, the number of quadrature points becomes too high. A possible solution can be found among the QuasiMonte Carlo (QMC) methods, because these techniques yield quite good approximations for high dimensional integrals with a much lower number of points, chosen for their optimal location. In this paper a comparison will be made between three integration methods: GH, QMC, and full Monte Carlo integration (MC).
Faster Quasi Random Number Generator for
, 2005
"... Generating quasirandom sequences for Monte Carlo simulations has been proved of interest when comparing results with numbers coming from conventional pseudorandom sequences. Despite a fair amount of sound theoretical work on random number generation algorithms, theoretical gain is often eradicated ..."
Abstract
 Add to MetaCart
(Show Context)
Generating quasirandom sequences for Monte Carlo simulations has been proved of interest when comparing results with numbers coming from conventional pseudorandom sequences. Despite a fair amount of sound theoretical work on random number generation algorithms, theoretical gain is often eradicated by poor computer implementations. Recently, one of the authors of this paper showed that massive unrolling optimizations could now be efficiently applied to pseudorandom number generation on regular desktop computers. This technique does not depend on the generation algorithm but it only stems from the classical unrolling optimization. In this paper we extend this technique to quasirandom generation and we present a memory mapping technique which can be efficiently associated to unrolling to overcome the limits we had with our previous unrolling technique for pseudo random numbers. Both techniques presented are portable on Unix derived platforms. Every research field using quasiMonte Carlo simulation can be concerned by this software optimization technique which is even more adapted to quasirandom numbers than to pseudorandom numbers.