Similarity-Based Estimation of Word Cooccurrence Probabilities

Similarity-Based Estimation of Word Cooccurrence Probabilities (1994)

Cached

Download Links

Other Repositories/Bibliography

DBLP

by Ido Dagan , Fernando Pereira , Lillian Lee

Venue:	In Proceedings of the 32nd Annual Meeting of the Association for Computational Linguistics
Citations:	65 - 7 self

Versions

Version History

Metadata Version 3

User correction supplied by SystemCorrections

Datum	Value	Source
TITLE	Similarity-Based Estimation of Word Cooccurrence Probabilities	user correction - Legacy Corrections
AUTHOR NAME	Ido Dagan	user correction - Legacy Corrections
AUTHOR AFFIL	AT&T Bell Laboratories	user correction - Legacy Corrections
AUTHOR ADDR	600 Mountain Ave.; Murray Hill, NJ 07974, USA	user correction - Legacy Corrections
AUTHOR NAME	Fernando Pereira	user correction - Legacy Corrections
AUTHOR AFFIL	AT&T Bell Laboratories	user correction - Legacy Corrections
AUTHOR ADDR	600 Mountain Ave.; Murray Hill, NJ 07974, USA	user correction - Legacy Corrections
AUTHOR NAME	Lillian Lee	user correction - Legacy Corrections
ABSTRACT	In many applications of natural language processing it is necessary to determine the likelihood of a given word combination. For example, a speech recognizer may need to determine which of the two word combinations "eat a peach" and "eat a beach" is more likely. Statistical NLP methods determine the likelihood of a word combination according to its frequency in a training corpus. However, the nature of language is such that many word combinations are infrequent and do not occur in a given corpus. In this work we propose a method for estimating the probability of such previously unseen word combinations using available information on "most sim- ilar" words. We describe a probabilistic word association model based on distributional word similarity, and apply it to improving probability estimates for unseen word bigrams in a variant of Katz's back-off model. The similarity-based method yields a 20% perplexity improvement in the prediction of unseen bigrams and statistically significant reductions in speech-recognition error.	user correction - Legacy Corrections
YEAR	1994	INFERENCE
VENUE	In Proceedings of the 32nd Annual Meeting of the Association for Computational Linguistics	INFERENCE
VENUE TYPE	CONFERENCE	INFERENCE
PAGES	272--278	INFERENCE
CITATIONS	14 found	ParsCit 1.0