Abstract

Abstract. Anyone offering content in a digital library is naturally interested in assessing its performance: how well does my system meet the users ’ information needs? Standard evaluation benchmarks have been developed in information retrieval that can be used to test retrieval effectiveness. However, these generic benchmarks focus on a single document genre, language, media-type, and searcher stereotype that is radically different from the unique content and user community of a particular digital library. This paper proposes to derive a domain-specific test collection from readily available interaction data in search logs files that captures the domain-specificity of digital libraries. We use an archival institution’s complete search log that spans over multiple years, and derive a large-scale test collection. We use this resulting domain specific test collection to evaluate a range of retrieval models. Finally, we manually derive a set of topics judged by human experts—based on a set of e-mail reference questions and responses from archivists—and use this for validation. Our main finding is that we can derive a massive, reliable and domain-specific test collection from search log files. 1

Citations