Abstract

ment retrieval have been extensively studied. The in-tegration of these two classes of models has been the goal of several researchers but it is a very difficult prob-lem. We argue that much of the reason for this is the lack of an adequate indexing model. This suggests that perhaps a better indexing model would help solve the problem. However, we feel that making unwarranted parametric assumptions will not lead to better retrieval performance. Furthermore, making prior assumptions about the similarity of documents is not warranted ei-ther. Instead, we propose an approach to retrieval based on probabilistic language modeling. We estimate models for each document individually. Our approach to model-ing is non-parametric and integrates document indexing and document retrieval into a single model. One advan-tage of our approach is that collection statistics which are used heuristically in many other retrieval models are an integral part of our model. We have implemented our model and tested it empirically. Our approach sig-nificantly outperforms standard tf.idf weighting on two different collections and query sets. 1

Citations

1574	Density Estimation for Statistics and Data Analysis Chapman - Silverman - 1986
510	Relevance weighting of search terms - Robertson, Jones - 1976
481	On estimation of a probability density function and mode - Parzen - 1967
289	Some simple effective approximations to the 2-poisson model for probabilistic weighted retrieval - Robertson, Walker - 1994
67	Text segmentation by topic - Ponte, Croft - 1997
45	Efficient probabilistic inference for text retrieval - Turtle, Croft - 1991
34	A new method of weighting query terms for ad-hoc retrieval - Kwok - 1996
31	A New Probabilistic Model of Text Classification and Retrieval - Kalt - 1996
28	An inquiry into the nature of frequency distributions representative of multiple happenings with particular reference to the occurrence of multiple attacks of disease or of repeated accidents - Greenwood, Yule - 1920
25	Oversmoothed Nonparametric Density Estimates - Terrell, Scott - 1985
18	Modelling documents with multiple Poisson distributions - Margulis - 1993
3	Automatic Tezt Processing - Salton - 1989
2	Topic Detection and Tracking Segmentation Task” In proceedings of The Topic Detection and Tracking Workshop - Yamron - 1997