Abstract:
This paper proposes a new method for evaluating the quality of retrieval functions. Unlike traditional methods that require relevance judgments by experts or explicit user feedback, it is based entirely on clickthrough data. This is a key advantage, since clickthrough data can be collected at very low cost and without overhead for the user. Taking an approach from experiment design, the paper proposes an experiment setup that generates unbiased feedback about the relative quality of two search results without explicit user feedback. A theoretical analysis shows that the method gives the same results as evaluation with traditional relevance judgments under mild assumptions. An empirical analysis veri es that the assumptions are indeed justi ed and that the new method leads to conclusive results in a WWW retrieval study.
Citations
|
1439
|
Modern Information Retrieval
– Baeza-Yates, Ribeiro
- 1999
|
|
915
|
Term-weighting approaches in automatic text retrieval
– Salton, Buckley
- 1988
|
|
468
|
An agent that assists web browsing
– Lieberman
- 1995
|
|
264
|
WebWatcher: A Tour Guide for the World Wide Web
– Joachims, Freitag, et al.
- 1997
|
|
262
|
Optimizing search engines using clickthrough data
– Joachims
- 2002
|
|
134
|
Analysis of a very large AltaVista query log
– Silverstein, Henzinger, et al.
- 1998
|
|
111
|
Automatic combination of multiple ranked retrieval systems
– Bartell, Cottrell, et al.
- 1994
|
|
92
|
Overview of the Eighth Text REtrieval Conference (TREC-8),” presented at Text REtrieval Conference (TREC-8
– Voorhees, Harman
|
|
71
|
Finding information on the World Wide Web: The retrieval effectiveness of search engines
– Gordon, Pathak
- 1999
|
|
60
|
A machine learning architecture for optimizing Web search engine
– Boyan, Freitag, et al.
- 1996
|
|
33
|
Ranking Retrieval Systems without Relevance Judgments
– Soboroff, Nicholas, et al.
- 2001
|
|
28
|
Optimum polynomial retrieval functions based on the probability ranking principle
– Fuhr
- 1989
|
|
26
|
First 20 precision among World Wide Web search services (Search Engines
– Leighton, Srivastava
- 1999
|
|
18
|
A field experimental approach to the study of relevance assessments in relation to document searching
– Rees, Schultz
- 1967
|
|
16
|
Introduction to the Theory of Statistics, McGraw-Hill Companies
– Mood, Graybill, et al.
- 1974
|
|
14
|
Report on the need for and provision of an "ideal" information retrieval test collection
– Jones, Rijsbergen
- 1975
|
|
12
|
Meta-scoring: automatically evaluating term weighting schemes in IR without precision-recall
– Jin, Falusos, et al.
- 2001
|
|
8
|
Relevance assessments and retrieval system evaluation. Information Storage and Retrieval
– Lesk, Salton
- 1969
|
|
8
|
A new method for automatic performance comparison of search engines
– Li, Shang
- 2000
|
|
2
|
The Melbourne TREC-9 Experiments
– D’Souza, Fuller
- 2000
|
|
2
|
Determining the eectiveness of retrieval algorithms
– Frei, Schauble
- 1991
|
|
1
|
Text Mining: Theoretical Aspects and Applications, chapter Evaluating Retrieval Performance using Clickthrough Data
– Joachims
- 2003
|
|
