Abstract

We study distributional similarity measures for the purpose of improving probability estimation for unseen cooccurrences. Our contributions are three-fold: an empirical comparison of a broad range of measures; a classification of similarity functions based on the information that they incorporate; and the introduction of a novel function that is superior at evaluating potential proxy distributions.

Citations

6515	Elements of Information Theory - Cover, Thomas - 1991
2699	MJ: Introduction to Modern Information Retrieval - Salton, McGill - 1986
649	A stochastic parts program and noun phrase parser for unrestricted text - Church - 1988
574	Estimation of probabilities from sparse data for the language model component of a speech recognizer - Katz - 1987
528	Automatic retrieval and clustering of similar words - Lin - 1998
526	Using Information Content to Evaluate Semantic Similarity in a Taxonomy - Resnik
439	Toward memory-based reasoning - Stanfill, Waltz - 1986
298	Divergence measures based on the Shannon entropy - Lin - 1991
286	Interpolated estimation of Markov source parameters from sparse data - Jelinek, Mercer - 1980
204	Integrating multiple knowledge sources to disambiguate word sense: An exemplar-based approach - Ng, Lee - 1996
198	Noun classification from predicate-argument structures - Hindle - 1990
120	Translating collocations for bilingual lexicons: A statistical approach - Smadja, McKeown, et al. - 1996
115	Text-translation alignment - Kay, Röscheisen - 1993
103	Using syntactic dependency as local context to resolve word sense ambiguity - Lin - 1997
73	Contextual word similarity and estimation from sparse data - Dagan, Markus, et al. - 1993
69	A Case-Based Approach to Knowledge Acquisition for DomainSpecific Sentence Analysis - Cardie - 1993
59	Towards the automatic identification of adjectival scales: Clustering adjectives according to meaning - Hatzivassiloglou, McKeown - 1993
48	Similarity-based word sense disambiguation - Karov, Edelman - 1998
43	Exemplar-based word sense disambiguation: Some recent improvements - Ng - 1997
40	Coocurrence smoothing for stochastic language modeling - Essen, Steinbiss - 1992
35	Statstical sense disambiguation with relatively small corpora using dictionary definitions - Luk - 1995
33	Similarity-based approaches to natural language processing - Lee - 1997
19	Smoothing of automatically generated selectional constraints - Grishman, Sterling - 1993
17	Diversity: Its Measurement, Decomposition, Apportionment and Analysis. Sankhy: The Indian - Rao - 1982
11	Do We Need Linguistics When We Have Statistics? A Comparative Analysis of the Contributions of Linguistic Cues to a Statistical Word Grouping System - Hatzivassiloglou - 1994
11	JD: Nonparametric Measures of Association - Gibbons
9	An information theoretic definition of similarity - Lin - 1998
9	Isolated word recognition using hidden Markov models - Sugawara, Nishimura, et al. - 1985
3	Nonparametric Measures of Association. Number 07-091 - Gibbons - 1993