Abstract

Two stages in measurement of techniques for information retrieval are gathering of documents for relevance assessment and use of the assessments to numerically evaluate e#ectiveness. We consider both of these stages in the context of the TREC experiments, to determine whether they lead to measurements that are trustworthy and fair. Our detailed empirical investigation of the TREC results shows that the measured relative performance of systems appears to be reliable, but that recall is overestimated: it is likely that many relevant documents have not been found. We propose a new pooling strategy that can significantly increase the number of relevant documents found for given e#ort, without compromising fairness.

Citations

142	P: Scientific reasoning: The Bayesian approach - Howson, Urbach
122	Overview of the second text retrieval conference - Harman - 1995
110	Overview of the fourth text retrieval conference (TREC-4 - Harman - 1995
89	Overview of the fifth Text REtrieval Conference (TREC-5 - Voorhees, Harman - 1996
69	The pragmatics of information retrieval experimentation, revisited - Tague-Sutcliffe - 1992
67	A critical investigation of recall and precision as measures of retrieval system performance - Raghavan, Bollmann, et al. - 1989
52	Variations in relevance assessments and the measurement of retrieval effectiveness - Harter - 1996
41	Retrieval of partial documents - Moffat, Sacks-Davis, et al. - 1993
38	Relevance assessment and retrieval system evaluation - Lesk, Salton - 1968
38	The state of retrieval system evaluation - Salton - 1992
31	A statistical analysis of the TREC-3 data - Tague-Sutcliffe, Blustein - 1994
30	Statistical Inference in Retrieval Effectiveness Evaluation - Savoy - 1997
8	The Cranfield II relevance assessments: A critical evaluation - Harter - 1971
8	Some unexplained aspects of the Cranfield tests of indexing performance factors. Ltbrary Quarterly - Swanson - 1971
4	redux: Thoughts on the STAIRS evaluation, ten years after - STAIRS - 1996
4	Relevance judgements for assessing recall - Wallis, Thom - 1996