Abstract

Techniques of exploratory data analysis are used to study the weight of evidence that the occurrence of a query term provides in support of the hypothesis that a document is relevant to an information need. In particular, the relationship between the document frequency and the weight of evidence is investigated. A correlation between document frequency normalized by collection size and the mutual information between relevance and term occurrence is uncovered. This correlation is found to be robust across a variety of query sets and document collections. Based on this relationship, a theoretical explanation of the efficacy of inverse document frequency for term weighting is developed which differs in both style and content from theories previously put forth. The theory predicts that a "flattening" of idf at both low and high frequency should result in improved retrieval performance. This altered idf formulation is tested on all TREC query sets. Retrieval results corroborate the predicti...

Citations

2699	Introduction to Modern Information Retrieval - Salton, McGill - 1983
639	Exploratory Data Analysis - Tukey - 1977
510	Relevance weighting of search terms - Robertson, Jones - 1976
309	The INQUERY retrieval system - Callan, Croft, et al. - 1992
168	The first text retrieval conference - Harman - 1993
151	Using probabilistic models of document retrieval without relevance information - Croft, Harper - 1979
131	Using statistics in lexical analysis - Church, Gale, et al. - 1991
100	Transmission of Information: A Statistical Theory of Communications - Fano - 1961
89	Overview of the fifth Text REtrieval Conference (TREC-5 - Voorhees, Harman - 1996
78	Probability and The Weighing of Evidence - Good - 1950
56	On relevance weights with little relevance information - Robertson, Walker - 1997
53	Full text retrieval based on probabilistic equaitions with coefficients fitted by logistic regression. The Second Text REtrieval Conference - Cooper, Chen, et al. - 1994
50	A statistical interpretation of term specificity and its applicatoin in retrieval - Sparck-Jones - 1972
49	An evaluation of feedback in document retrieval using cooccurrence data - Harper, Rijsbergen - 1978
39	Inferring probability of relevance using the method of logistic regression - Gey - 1994
37	Probabilistic Retrieval Based on Staged Logistic Regression - Cooper, Gey, et al. - 1992
32	Optimum polynomial retrieval functions based on the probability ranking principle - Fuhr - 1989
19	Corpus-specific stemming using word form cooccurence - Croft, Xu - 1995
19	Probabilistic document indexing from relevance feedback data - FUHR, BUCKLEY - 1991
19	Weight of evidence: a brief survey - Good - 1985
16	Exploratory Data Analysis - Hartwig, Dearing - 1979
12	Term Weighting in Information Retrieval Using the Term Precision Model - Yu, Lam, et al. - 1982
11	Declarative specifications - Fuchs, Robertson - 1996
9	Automatic indexing using term discrimination and term precision measurements - Salton, Wong, et al. - 1976
9	TheMeasurement of TermImportance in Automatic Indexing - Salton, Wu, et al. - 1981
8	An information-theoretic measure of term specificity - Wong, Yao - 1992
7	Two learning schemes in information retrieval - Yu, Mizuno - 1988