We address the problem of distinguishing between two sources of disagreement in annotations: genuine subjectivity and slip of attention. The latter is especially likely when the classification task has a default class, as in tasks where annotators need to find instances of the phenomenon of interest, such as in a metaphor detection task discussed here. We apply and extend a data analysis technique proposed by Beigman Klebanov and Shamir (2006) to first distill reliably deliberate (non-chance) annotations and then to estimate the amount of attention slips vs genuine disagreement in the reliably deliberate annotations. 1

Abstract. Lexical cohesion is an important device for signalling text organisation. In this paper, we investigate to what extent idiomatic expressions participate in the cohesive structure of a text. We look at the problem both from an empirical and a computational perspective. Our results show that both literal and nonliteral expressions exhibit cohesion with their context, though the latter tend to do so to a lesser extent. We also show that cohesive links identified by humans can be approximated by an automatically computable semantic relatedness measure based on search engine page counts.

This technical report describes in some detail (1) the creation of a dataset for testing the degree of relatedness between concepts out of the data from Beigman Klebanov and Shamir's lexical cohesion experiment [3, 5, 6], and (2) a new measure of semantic relatedness based on WordNet. We welcome comments on this manuscript; however, please refrain from citing it, but rather the concise published version [4]. This report is intended to accompany the published paper with more thorough technical detail to enable replication of the method.