DMCA
How Reliable are the Results of Large-Scale Information Retrieval Experiments? (1998)
Cached
Download Links
- [www.cs.rmit.edu.au]
- [www.seg.rmit.edu.au]
- [goanna.cs.rmit.edu.au]
- [people.eng.unimelb.edu.au]
- DBLP
Other Repositories/Bibliography
| Venue: | Proceedings of the 21st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval |
| Citations: | 156 - 4 self |
BibTeX
@INPROCEEDINGS{Zobel98howreliable,
author = {Justin Zobel},
title = {How Reliable are the Results of Large-Scale Information Retrieval Experiments?},
booktitle = {Proceedings of the 21st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval},
year = {1998},
pages = {307--314},
publisher = {ACM Press}
}
Share
 |
 |
 |
 |
||
OpenURL
Abstract
Two stages in measurement of techniques for information retrieval are gathering of documents for relevance assessment and use of the assessments to numerically evaluate e#ectiveness. We consider both of these stages in the context of the TREC experiments, to determine whether they lead to measurements that are trustworthy and fair. Our detailed empirical investigation of the TREC results shows that the measured relative performance of systems appears to be reliable, but that recall is overestimated: it is likely that many relevant documents have not been found. We propose a new pooling strategy that can significantly increase the number of relevant documents found for given e#ort, without compromising fairness.
