A Search Log-Based Approach to Evaluation
| Citations: | 3 - 1 self |
BibTeX
@MISC{Zhang_asearch,
author = {Junte Zhang and Jaap Kamps},
title = {A Search Log-Based Approach to Evaluation},
year = {}
}
OpenURL
Abstract
Abstract. Anyone offering content in a digital library is naturally interested in assessing its performance: how well does my system meet the users ’ information needs? Standard evaluation benchmarks have been developed in information retrieval that can be used to test retrieval effectiveness. However, these generic benchmarks focus on a single document genre, language, media-type, and searcher stereotype that is radically different from the unique content and user community of a particular digital library. This paper proposes to derive a domain-specific test collection from readily available interaction data in search logs files that captures the domain-specificity of digital libraries. We use an archival institution’s complete search log that spans over multiple years, and derive a large-scale test collection. We use this resulting domain specific test collection to evaluate a range of retrieval models. Finally, we manually derive a set of topics judged by human experts—based on a set of e-mail reference questions and responses from archivists—and use this for validation. Our main finding is that we can derive a massive, reliable and domain-specific test collection from search log files. 1







