Citation Context

...t derivative of the logarithm of the gamma function). The updates in equation 5 are guaranteed to converge to a stationary point of the ELBO. By analogy to the Expectation-Maximization (EM) algorithm =-=[14]-=-, we can partition these updates into an “E” step—iteratively updating γ and φ until convergence, holding λ fixed—and an “M” step—updating λ given φ. In practice, this algorithm converges to a better ...

Citation Context

...onal Bayes is a practical new method for estimating the posterior of complex hierarchical Bayesian models. 2 Online variational Bayes for latent Dirichlet allocation Latent Dirichlet Allocation (LDA) =-=[7]-=- is a Bayesian probabilistic model of text documents. It assumes a collection of K “topics.” Each topic defines a multinomial distribution over the vocabulary and is assumed to have been drawn from a ...

Citation Context

...ction. 2.1 Batch variational Bayes for LDA In Variational Bayesian inference (VB) the true posterior is approximated by a simpler distribution q(z, θ, β), which is indexed by a set of free parameters =-=[12, 13]-=-. These parameters are optimized to maximize the Evidence Lower BOund (ELBO): log p(w|α, η) ≥L(w, φ, γ, λ) � Eq[log p(w, z, θ, β|α, η)] − Eq[log q(z, θ, β)]. (1) Maximizing the ELBO is equivalent to m...

Citation Context

...pling (CGS) is a popular MCMC approach that samples from the posterior over topic assignments z by repeatedly sampling the topic assignment zdi conditioned on the data and all other topic assignments =-=[22]-=-. One online MCMC approach adapts CGS by sampling topic assignments zdi based on the topic assignments and data for all previously analyzed words, instead of all other words in the corpus [23]. This a...

Citation Context

... fit a set of per-observation parameters (such as the per-document variational parameters γd and φd in LDA). The problem is addressed by online coordinate ascent algorithms such as those described in =-=[20, 21, 16, 17, 10]-=-. The goal of these algorithms is to set the global parameters so that the objective is as good as possible once the perobservation parameters are optimized. Most of these approaches assume the comput...

Citation Context

...laces sampling with optimization, we can use results from stochastic optimization to analyze online LDA. Stochastic optimization algorithms optimize an objective using noisy estimates of its gradient =-=[18]-=-. Although there is no explicit gradient computation, algorithm 2 can be interpreted as a stochastic natural gradient algorithm [16, 15]. We begin by deriving a related first-order stochastic gradient...

Citation Context

...ars in document d) into a matrix of topic weights θ and a dictionary of topics β [9]. Our work can thus 2be seen as an extension of online matrix factorization techniques that optimize squared error =-=[10]-=- to more general probabilistic formulations. We can analyze a corpus of documents with LDA by examining the posterior distribution of the topics β, topic proportions θ, and topic assignments z conditi...

Citation Context

...h many others are based. Our algorithm is based on online stochastic optimization, which has been shown to produce good parameter estimates dramatically faster than batch algorithms on large datasets =-=[6]-=-. Online LDA handily analyzes massive collections of documents and, moreover, online LDA need not locally store or collect the documents— each can arrive in a stream and be discarded after one look. I...

Citation Context

...ction. 2.1 Batch variational Bayes for LDA In Variational Bayesian inference (VB) the true posterior is approximated by a simpler distribution q(z, θ, β), which is indexed by a set of free parameters =-=[12, 13]-=-. These parameters are optimized to maximize the Evidence Lower BOund (ELBO): log p(w|α, η) ≥L(w, φ, γ, λ) � Eq[log p(w, z, θ, β|α, η)] − Eq[log q(z, θ, β)]. (1) Maximizing the ELBO is equivalent to m...

Citation Context

...ld fixed. In all experiments α and η are fixed at 0.01 and the number of topics K = 100. There is some question as to the meaningfulness of perplexity as a metric for comparing different topic models =-=[25]-=-. Held-out likelihood metrics are nonetheless well suited to measuring how well an inference algorithm accomplishes the specific optimization task defined by a model. Evaluating learning parameters. O...

Citation Context

...choice of approximate posterior introduces bias, VB is empirically shown to be faster than and as accurate as MCMC, which makes it an attractive option when applying Bayesian models to large datasets =-=[1, 2, 3]-=-. Nonetheless, large scale data analysis with VB can be computationally difficult. Standard “batch” VB algorithms iterate between analyzing each observation and updating dataset-wide variational param...

Citation Context

... fit a set of per-observation parameters (such as the per-document variational parameters γd and φd in LDA). The problem is addressed by online coordinate ascent algorithms such as those described in =-=[20, 21, 16, 17, 10]-=-. The goal of these algorithms is to set the global parameters so that the objective is as good as possible once the perobservation parameters are optimized. Most of these approaches assume the comput...

Citation Context

...ments. to summarize the latent structure of massive document collections that cannot be annotated by hand. A central research problem for topic modeling is to efficiently fit models to larger corpora =-=[4, 5]-=-. To this end, we develop an online variational Bayes algorithm for latent Dirichlet allocation (LDA), one of the simplest topic models and one on which many others are based. Our algorithm is based o...

Citation Context

... are effective, but both present significant computational challenges in the face of massive data sets.Developing scalable approximate inference methods for topic models is an active area of research =-=[3, 4, 5, 11]-=-. To this end, we develop online variational inference for LDA, an approximate posterior inference algorithm that can analyze massive collections of documents. We first review the traditional variatio...

Citation Context

...rithm. The condition that κ ∈ (0.5, 1] is needed to guarantee convergence. We show in section 2.3 that online LDA corresponds to a stochastic natural gradient algorithm on the variational objective L =-=[15, 16]-=-. This algorithm closely resembles one proposed in [16] for online VB on models with hidden data— the most important difference is that we use an approximate E step to optimize γt and φt, since we can...

Citation Context

...choice of approximate posterior introduces bias, VB is empirically shown to be faster than and as accurate as MCMC, which makes it an attractive option when applying Bayesian models to large datasets =-=[1, 2, 3]-=-. Nonetheless, large scale data analysis with VB can be computationally difficult. Standard “batch” VB algorithms iterate between analyzing each observation and updating dataset-wide variational param...

Citation Context

...er words in the corpus [23]. This algorithm is fast and has constant memory requirements, but is not guaranteed to converge to the posterior. Two alternative online MCMC approaches were considered in =-=[24]-=-. The first, called incremental LDA, periodically resamples the topic assignments for previously analyzed words. The second approach uses particle filtering instead of CGS. In a study in [24], none of...

Citation Context

...gnments [22]. One online MCMC approach adapts CGS by sampling topic assignments zdi based on the topic assignments and data for all previously analyzed words, instead of all other words in the corpus =-=[23]-=-. This algorithm is fast and has constant memory requirements, but is not guaranteed to converge to the posterior. Two alternative online MCMC approaches were considered in [24]. The first, called inc...

Citation Context

...ments. to summarize the latent structure of massive document collections that cannot be annotated by hand. A central research problem for topic modeling is to efficiently fit models to larger corpora =-=[4, 5]-=-. To this end, we develop an online variational Bayes algorithm for latent Dirichlet allocation (LDA), one of the simplest topic models and one on which many others are based. Our algorithm is based o...

Citation Context

...choice of approximate posterior introduces bias, VB is empirically shown to be faster than and as accurate as MCMC, which makes it an attractive option when applying Bayesian models to large datasets =-=[1, 2, 3]-=-. Nonetheless, large scale data analysis with VB can be computationally difficult. Standard “batch” VB algorithms iterate between analyzing each observation and updating dataset-wide variational param...

Citation Context

...d φt as random variables ∑drawn at the same time as each observed document nt, then Eg[D∇λℓ(nt, γt, φt, λ)|λ] = ∇λ d ℓ(nd, γd, φd, λ). Thus, since ∑∞ t=0 ρt = ∞ and ∑∞ t=0 ρ2 ∑ t < ∞, the analysis in =-=[19]-=- shows both that λ converges and that the gradient ∇λ d ℓ(nd, γd, φd, λ) converges to 0, and thus that λ converges to a stationary point. 1 The update in equation 12 only makes use of first-order grad...

Citation Context

...rithm. The condition that κ ∈ (0.5, 1] is needed to guarantee convergence. We show in section 2.3 that online LDA corresponds to a stochastic natural gradient algorithm on the variational objective L =-=[15, 16]-=-. This algorithm closely resembles one proposed in [16] for online VB on models with hidden data— the most important difference is that we use an approximate E step to optimize γt and φt, since we can...

Citation Context

...nk of LDA as a probabilistic factorization of the matrix of word counts n (where ndw is the number of times word w appears in document d) into a matrix of topic weights θ and a dictionary of topics β =-=[9]-=-. Our work can thus 2be seen as an extension of online matrix factorization techniques that optimize squared error [10] to more general probabilistic formulations. We can analyze a corpus of document...