DMCA
Cumulated Gain-based Evaluation of IR Techniques (2002)
| Venue: | ACM Transactions on Information Systems |
| Citations: | 692 - 3 self |
BibTeX
@ARTICLE{Järvelin02cumulatedgain-based,
author = {Kalervo Järvelin and Jaana Kekäläinen},
title = {Cumulated Gain-based Evaluation of IR Techniques},
journal = {ACM Transactions on Information Systems},
year = {2002},
volume = {20},
pages = {2002}
}
Share
 |
 |
 |
 |
||
OpenURL
Abstract
Modem large retrieval environments tend to overwhelm their users by their large output. Since all documents are not of equal relevance to their users, highly relevant documents should be identified and ranked first for presentation to the users. In order to develop IR techniques to this direction, it is necessary to develop evaluation approaches and methods that credit IR methods for their ability to retrieve highly relevant documents. This can be done by extending traditional evaluation methods, i.e., recall and precision based on binary relevance assessments, to graded relevance assessments. Alternatively, novel measures based on graded relevance assessments may be developed. This paper proposes three novel measures that compute the cumulative gain the user obtains by examining the retrieval result up to a given ranked position. The first one accumulates the relevance scores of retrieved documents along the ranked result list. The second one is similar but applies a discount factor on the relevance scores in order to devaluate late-retrieved documents. The third one computes the relative-tothe -ideal performance of IR techniques, based on the cumulative gain they are able to yield. The novel measures are defined and discussed and then their use is demonstrated in a case study using TREC data - sample system run results for 20 queries in TREC-7. As relevance base we used novel graded relevance assessments on a four-point scale. The test results indicate that the proposed measures credit IR methods for their ability to retrieve highly relevant documents and allow testing of statistical significance of effectiveness differences. The graphs based on the measures also provide insight into the performance IR techniques and allow interpretation, e.g., from the user point of ...
Keyphrases
ir technique gain-based evaluation relevant document novel measure cumulative gain relevance score graded relevance assessment retrieved document user obtains discount factor relevance assessment traditional evaluation method modem large retrieval environment equal relevance ranked position statistical significance allow testing relevance base four-point scale case study measure credit ir method evaluation approach credit ir method binary relevance assessment trec data sample system relative-tothe ideal performance large output test result ranked result list late-retrieved document user point second one performance ir technique retrieval result effectiveness difference
