Citation Context

...ons. 2.2 Image Modelling: the Conditional alignDRAW Network To generate images conditioned on the caption information, we extended the DRAW network [1] to include caption representation hlang at each step, shown in Figure 1. The conditional DRAW network is a stochastic recurrent neural network that consists of a set of latent variables Zt at each time step. Unlike the original DRAW network where latent variables are independent unit Gaussians N (0, I), the latent variables in the proposed alignDRAW model have their mean and variance depend on the previous recurrent hidden states hdect−1 as in [4]. Formally, the image is generated by iteratively computing the following equations for t = 1, ..., T (see Figure 1): zt ∼ p(Zt|Z1:t−1) = N (µt(hdect−1), σt(hdect−1)), (1) hdect = LSTM dec(hdect−1, zt, st−1), (2) st = align(h dec t−1,h lang); ct = ct−1 + write(h dec t ), (3) where write and read are the same attention operators as in [1]. The align function is used to compute the alignment between the input caption and intermediate image generative steps [5]. Given the caption representation from the language model, hlang = [hlang1 , h lang 2 , ..., h lang N ], the align operator outputs a dyn...