A General Language Model for Information Retrieval

DMCA

A General Language Model for Information Retrieval (1999)

Cached

Download Links

Other Repositories/Bibliography

DBLP

by Fei Song , W. Bruce Croft

Venue:	In Proceedings of the 1999 ACM SIGIR Conference on Research and Development in Information Retrieval
Citations:	243 - 15 self

BibTeX

@INPROCEEDINGS{Song99ageneral,
    author = {Fei Song and W. Bruce Croft},
    title = {A General Language Model for Information Retrieval},
    booktitle = {In Proceedings of the 1999 ACM SIGIR Conference on Research and Development in Information Retrieval},
    year = {1999},
    pages = {279--280}
}

OpenURL

Abstract

Statistical language modeling has been successfully used for speech recognition, part-of-speech tagging, and syntactic parsing. Recently, it has also been applied to information retrieval. According to this new paradigm, each document is viewed as a language sample, and a query as a generation process. The retrieved documents are ranked based on the probabilities of producing a query from the corresponding language models of these documents. In this paper, we will present a new language model for information retrieval, which is based on a range of data smoothing techniques, including the Good-Turing estimate, curve-fitting functions, and model combinations. Our model is conceptually simple and intuitive, and can be easily extended to incorporate probabilities of phrases such as word pairs and word triples. The experiments with the Wall Street Journal and TREC4 data sets showed that the performance of our model is comparable to that of INQUERY and better than that of another language model for information retrieval. In particular, word pairs are shown to be useful in improving the retrieval performance.

Keyphrases

information retrieval general language model word pair curve-fitting function speech recognition good-turing estimate corresponding language model new language model model combination retrieved document syntactic parsing language sample part-of-speech tagging trec4 data set statistical language modeling generation process word triple retrieval performance language model wall street journal new paradigm

DMCA