Results 1 - 10
of
86
A convolutional neural network for modelling sentences.
- In Proceedings of the 52th Annual Meeting of the Association for Computational Linguistics.
, 2014
"... Abstract The ability to accurately represent sentences is central to language understanding. We describe a convolutional architecture dubbed the Dynamic Convolutional Neural Network (DCNN) that we adopt for the semantic modelling of sentences. The network uses Dynamic k-Max Pooling, a global poolin ..."
Abstract
-
Cited by 59 (2 self)
- Add to MetaCart
(Show Context)
Abstract The ability to accurately represent sentences is central to language understanding. We describe a convolutional architecture dubbed the Dynamic Convolutional Neural Network (DCNN) that we adopt for the semantic modelling of sentences. The network uses Dynamic k-Max Pooling, a global pooling operation over linear sequences. The network handles input sentences of varying length and induces a feature graph over the sentence that is capable of explicitly capturing short and long-range relations. The network does not rely on a parse tree and is easily applicable to any language. We test the DCNN in four experiments: small scale binary and multi-class sentiment prediction, six-way question classification and Twitter sentiment prediction by distant supervision. The network achieves excellent performance in the first three tasks and a greater than 25% error reduction in the last task with respect to the strongest baseline.
Multistep regression learning for compositional distributional semantics
, 2013
"... We present a model for compositional distributional semantics related to the framework of Coecke et al. (2010), and emulating formal semantics by representing functions as tensors and arguments as vectors. We introduce a new learning method for tensors, generalising the approach of Baroni and Zampar ..."
Abstract
-
Cited by 29 (12 self)
- Add to MetaCart
(Show Context)
We present a model for compositional distributional semantics related to the framework of Coecke et al. (2010), and emulating formal semantics by representing functions as tensors and arguments as vectors. We introduce a new learning method for tensors, generalising the approach of Baroni and Zamparelli (2010). We evaluate it on two benchmark data sets, and find it to outperform existing leading methods. We argue in our analysis that the nature of this learning method also renders it suitable for solving more subtle problems compositional distributional models might face. 1
Concrete Sentence Spaces for Compositional Distributional Models of Meaning
"... Coecke, Sadrzadeh, and Clark [3] developed a compositional model of meaning for distributional semantics, in which each word in a sentence has a meaning vector and the distributional meaning of the sentence is a function of the tensor products of the word vectors. Abstractly speaking, this function ..."
Abstract
-
Cited by 25 (1 self)
- Add to MetaCart
(Show Context)
Coecke, Sadrzadeh, and Clark [3] developed a compositional model of meaning for distributional semantics, in which each word in a sentence has a meaning vector and the distributional meaning of the sentence is a function of the tensor products of the word vectors. Abstractly speaking, this function is the morphism corresponding to the grammatical structure of the sentence in the category of finite dimensional vector spaces. In this paper, we provide a concrete method for implementing this linear meaning map, by constructing a corpus-based vector space for the type of sentence. Our construction method is based on structured vector spaces whereby meaning vectors of all sentences, regardless of their grammatical structure, live in the same vector space. Our proposed sentence space is the tensor product of two noun spaces, in which the basis vectors are pairs of words each augmented with a grammatical role. This enables us to compare meanings of sentences by simply taking the inner product of their vectors. 1
Multilingual distributed representations without word alignment. ICLR
, 2014
"... Distributed representations of meaning are a natural way to encode covariance relationships between words and phrases in NLP. By overcoming data sparsity problems, as well as providing information about semantic relatedness which is not available in discrete representations, distributed representati ..."
Abstract
-
Cited by 11 (1 self)
- Add to MetaCart
(Show Context)
Distributed representations of meaning are a natural way to encode covariance relationships between words and phrases in NLP. By overcoming data sparsity problems, as well as providing information about semantic relatedness which is not available in discrete representations, distributed representations have proven useful in many NLP tasks. Recent work has shown how compositional semantic representations can successfully be applied to a number of monolingual applica-tions such as sentiment analysis. At the same time, there has been some initial success in work on learning shared word-level representations across languages. We combine these two approaches by proposing a method for learning distributed representations in a multilingual setup. Our model learns to assign similar embed-dings to aligned sentences and dissimilar ones to sentence which are not aligned while not requiring word alignments. We show that our representations are seman-tically informative and apply them to a cross-lingual document classification task where we outperform the previous state of the art. Further, by employing parallel corpora of multiple language pairs we find that our model learns representations that capture semantic relationships across languages for which no parallel data was used. 1
Towards a formal distributional semantics: Simulating logical calculi with tensors.
- Proceedings of the Second Joint Conference on Lexical and Computational Semantics.
, 2013
"... Abstract The development of compositional distributional models of semantics reconciling the empirical aspects of distributional semantics with the compositional aspects of formal semantics is a popular topic in the contemporary literature. This paper seeks to bring this reconciliation one step fur ..."
Abstract
-
Cited by 10 (1 self)
- Add to MetaCart
(Show Context)
Abstract The development of compositional distributional models of semantics reconciling the empirical aspects of distributional semantics with the compositional aspects of formal semantics is a popular topic in the contemporary literature. This paper seeks to bring this reconciliation one step further by showing how the mathematical constructs commonly used in compositional distributional models, such as tensors and matrices, can be used to simulate different aspects of predicate logic. This paper discusses how the canonical isomorphism between tensors and multilinear maps can be exploited to simulate a full-blown quantifier-free predicate calculus using tensors. It provides tensor interpretations of the set of logical connectives required to model propositional calculi. It suggests a variant of these tensor calculi capable of modelling quantifiers, using few non-linear operations. It finally discusses the relation between these variants, and how this relation should constitute the subject of future work.
Compositional-ly Derived Representations of Morphologically Complex Words in Distributional Semantics
"... Speakers of a language can construct an unlimited number of new words through morphological derivation. This is a major cause of data sparseness for corpus-based approaches to lexical semantics, such as distributional semantic models of word meaning. We adapt compositional methods originally develop ..."
Abstract
-
Cited by 10 (1 self)
- Add to MetaCart
(Show Context)
Speakers of a language can construct an unlimited number of new words through morphological derivation. This is a major cause of data sparseness for corpus-based approaches to lexical semantics, such as distributional semantic models of word meaning. We adapt compositional methods originally developed for phrases to the task of deriving the distributional meaning of morphologically complex words from their parts. Semantic representations constructed in this way beat a strong baseline and can be of higher quality than representations directly constructed from corpus data. Our results constitute a novel evaluation of the proposed composition methods, in which the full additive model achieves the best performance, and demonstrate the usefulness of a compositional morphology component in distributional semantics. 1
Recursive neural networks can learn logical semantics
- In Proc. of the 3rd Workshop on Continuous Vector Space Models and their Compositionality
, 2015
"... Tree-structured recursive neural networks (TreeRNNs) for sentence meaning have been successful for many applications, but it remains an open question whether the fixed-length representations that they learn can support tasks as demanding as logi-cal deduction. We pursue this question by evaluating w ..."
Abstract
-
Cited by 10 (4 self)
- Add to MetaCart
(Show Context)
Tree-structured recursive neural networks (TreeRNNs) for sentence meaning have been successful for many applications, but it remains an open question whether the fixed-length representations that they learn can support tasks as demanding as logi-cal deduction. We pursue this question by evaluating whether two such models— plain TreeRNNs and tree-structured neural tensor networks (TreeRNTNs)—can cor-rectly learn to identify logical relation-ships such as entailment and contradiction using these representations. In our first set of experiments, we generate artificial data from a logical grammar and use it to eval-uate the models ’ ability to learn to handle basic relational reasoning, recursive struc-tures, and quantification. We then evaluate the models on the more natural SICK chal-lenge data. Both models perform compet-itively on the SICK data and generalize well in all three experiments on simulated data, suggesting that they can learn suit-able representations for logical inference in natural language.
Evaluating Neural Word Representations in Tensor-Based Compositional Settings
"... We provide a comparative study be-tween neural word representations and traditional vector spaces based on co-occurrence counts, in a number of com-positional tasks. We use three differ-ent semantic spaces and implement seven tensor-based compositional models, which we then test (together with simpl ..."
Abstract
-
Cited by 9 (3 self)
- Add to MetaCart
We provide a comparative study be-tween neural word representations and traditional vector spaces based on co-occurrence counts, in a number of com-positional tasks. We use three differ-ent semantic spaces and implement seven tensor-based compositional models, which we then test (together with simpler ad-ditive and multiplicative approaches) in tasks involving verb disambiguation and sentence similarity. To check their scala-bility, we additionally evaluate the spaces using simple compositional methods on larger-scale tasks with less constrained language: paraphrase detection and di-alogue act tagging. In the more con-strained tasks, co-occurrence vectors are competitive, although choice of composi-tional method is important; on the larger-scale tasks, they are outperformed by neu-ral word embeddings, which show robust, stable performance across the tasks.
2012. Space projections as distributional models for semantic composition
- In In Proceedings of CICLing 2012
"... Abstract. Empirical distributional methods account for the meaning of syntactic structures by combining words according to algebraic operators (e.g. tensor product) acting over the corresponding lexical constituents. In this paper, a novel approach for semantic composition based on space projection ..."
Abstract
-
Cited by 8 (5 self)
- Add to MetaCart
(Show Context)
Abstract. Empirical distributional methods account for the meaning of syntactic structures by combining words according to algebraic operators (e.g. tensor product) acting over the corresponding lexical constituents. In this paper, a novel approach for semantic composition based on space projection techniques over the basic geometric lexical representations is proposed. In line with Frege’s context principle, the meaning of a phrase is modeled in terms of the subset of properties shared by the co-occurring words. In the geometric perspective here pursued, syntactic bi-grams are projected in the so called Support Subspace, aimed at emphasizing the semantic features shared by the compound words and better capturing phrase-specific aspects of the involved lexical meanings. State-of-the-art results are achieved in a well known phrase similarity task, used as a benchmark for this class of methods. 1
DISSECT- DIStributional SEmantics Composition Toolkit
"... We introduce DISSECT, a toolkit to build and explore computational models of word, phrase and sentence meaning based on the principles of distributional semantics. The toolkit focuses in particular on compositional meaning, and implements a number of composition methods that have been proposed in th ..."
Abstract
-
Cited by 8 (0 self)
- Add to MetaCart
We introduce DISSECT, a toolkit to build and explore computational models of word, phrase and sentence meaning based on the principles of distributional semantics. The toolkit focuses in particular on compositional meaning, and implements a number of composition methods that have been proposed in the literature. Furthermore, DISSECT can be useful to researchers and practitioners who need models of word meaning (without composition) as well, as it supports various methods to construct distributional semantic spaces, assessing similarity and even evaluating against benchmarks, that are independent of the composition infrastructure. 1
