Citation Context

...f these rely on a curated source of semantic information for the labels: the WordNet hierarchy is used in [12] and [17], and [16] uses a knowledge base containing descriptive properties for each class. By contrast, our approach learns its semantic representation directly from unannotated data. 3 Proposed Approach Our objective is to leverage semantic knowledge learned in the text domain, and transfer it to a model trained for visual object recognition. We begin by pre-training a simple neural language model wellsuited for learning semantically-meaningful, dense vector representations of words [13]. In parallel, we pre-train a state-of-the-art deep neural network for visual object recognition [11], complete with a traditional softmax output layer. We then construct a deep visual-semantic model by taking the lower layers of the pre-trained visual object recognition network and re-training them to predict the vector representation of the image label text as learned by the language model. These three training phases are detailed below. 3.1 Language Model Pre-training The skip-gram text modeling architecture introduced by Mikolov et al [13, 14] has been shown to efficiently learn semantical...

Citation Context

...learns a meaningful “semantic” representation of the entities and predicates, we can compute the nearest neighbors of various items in the a K-dimensional space. It is known from previous work (e.g., =-=[27]-=-) that related entities cluster together in the space, so here we focus on predicates. The results are shown in Table 4. We see that the model learns to put semantically related (but not necessarily s...

Citation Context

...f preserved variance, etc.). Occasionally, some kind of indirect supervision is used: Several parameter settings are tried, and the best setting is chosen based on performance on a semantic task that has been selected for tuning. The last few years have seen the development of a new generation of DSMs that frame the vector estimation problem directly as a supervised task, where the weights in a word vector are set to maximize the probability of the contexts in which the word is observed in the corpus (Bengio et al., 2003; Collobert and Weston, 2008; Collobert et al., 2011; Huang et al., 2012; Mikolov et al., 2013a; Turian et al., 2010). The traditional construction of context vectors is turned on its head: Instead of first collecting context vectors and then reweighting these vectors based on various criteria, the vector weights are directly set to optimally predict the contexts in which the corresponding words tend to appear. Since similar words occur in similar contexts, the system naturally learns to assign similar vectors to similar words. This new way to train DSMs is attractive because it replaces the essentially heuristic stacking of vector transforms in earlier models with a single, well-defin...

Citation Context

...ese vectors is high for semantically similar words. Several models have been proposed based on feedforward networks (Bengio et al., 2003), log-bilinear models (Mnih & Hinton, 2007), skip-gram models (=-=Mikolov et al., 2013-=-) and recurrent neural networks (Mikolov et al., 2010; 2011). Training can be sped up through the use of hierarchical softmax (Morin & Bengio, 2005) or noise contrastive estimation (Mnih & Teh, 2012)....

Citation Context

...mage embedding x, and vice-versa with xk. For all of our experiments, we initialize the word embeddings WT to be pre-computed K = 300 dimensional vectors learned using a continuous bag-of-words model =-=[37]-=-. The contrastive terms are chosen randomly from the training set and resampled every epoch. 1For additional details on LSTM: http://people.idsia.ch/~juergen/rnn.html. 2As a slight abuse of notation, ...

Citation Context

...s the entire space of activations, rather than the individual units, that contains the bulk of the semantic information. A similar, but even stronger conclusion was reached recently by Mikolov et al. =-=[12]-=- for word representations, where the various directions in the vector space representing the words are shown to give rise to a surprisingly rich semantic encoding of relations and analogies. At the sa...