DMCA
SimLex-999: Evaluating Semantic Models With (Genuine) Similarity Estimation
| Citations: | 9 - 3 self |
BibTeX
@MISC{Hill_simlex-999:evaluating,
author = {Felix Hill and Roi Reichart and Anna Korhonen},
title = {SimLex-999: Evaluating Semantic Models With (Genuine) Similarity Estimation},
year = {}
}
Share
 |
 |
 |
 |
||
OpenURL
Abstract
We present SimLex-999, a gold standard re-source for evaluating distributional semantic models that improves on existing resources in several important ways. First, in contrast to gold standards such as WordSim-353 and MEN, it explicitly quantifies similarity rather than association or relatedness so that pairs of entities that are associated but not actually similar (Freud, psychology) have a low rat-ing. We show that, via this focus on similar-ity, SimLex-999 incentivizes the development of models with a different, and arguably wider range of applications than those which reflect conceptual association. Second, SimLex-999 contains a range of concrete and abstract ad-jective, noun and verb pairs, together with an independent rating of concreteness and (free) association strength for each pair. This diver-sity enables fine-grained analyses of the per-formance of models on concepts of different types, and consequently greater insight into how architectures can be improved. Further, unlike existing gold standard evaluations, for which automatic approaches have reached or surpassed the inter-annotator agreement ceil-ing, state-of-the-art models perform well be-low this ceiling on SimLex-999. There is therefore plenty of scope for SimLex-999 to quantify future improvements to distributional semantic models, guiding the development of the next generation of representation-learning architectures. 1
Keyphrases
evaluating semantic model similarity estimation distributional semantic model gold standard evaluation several important way independent rating next generation gold standard re-source present simlex-999 conceptual association different type association strength low rat-ing verb pair fine-grained analysis representation-learning architecture automatic approach future improvement state-of-the-art model inter-annotator agreement ceil-ing
