LETOR: A Benchmark Collection for Research on Learning to Rank for Information Retrieval

LETOR: A Benchmark Collection for Research on Learning to Rank for Information Retrieval

Cached

Download Links

[research.microsoft.com]

by Tao Qin , Tie-yan Liu , Jun Xu , Hang Li

Citations:

11 - 1 self

BibTeX

@MISC{Qin_letor:a,
    author = {Tao Qin and Tie-yan Liu and Jun Xu and Hang Li},
    title = { LETOR: A Benchmark Collection for Research on Learning to Rank for Information Retrieval},
    year = {}
}

Bookmark

OpenURL

Abstract

LETOR is a benchmark collection for the research on learning to rank for information retrieval, released by Microsoft Research Asia. In this paper, we describe the details of the LETOR collection and show how it can be used in different kinds of researches. Specifically, we describe how the document corpora and query sets in LETOR are selected, how the documents are sampled, how the learning features and meta information are extracted, and how the datasets are partitioned for comprehensive evaluation. We then compare several state-of-the-art learning to rank algorithms on LETOR, report their ranking performances, and make discussions on the results. After that, we discuss possible new research topics that can be supported by LETOR, in addition to algorithm comparison. We hope that this paper can help people to gain deeper understanding of LETOR, and enable more interesting research projects on learning to rank and related topics.

Citations

2569	The Anatomy of a Large-Scale Hypertextual Web Search Engine - Brin, Page - 1998
1926	Modern Information Retrieval - Baeza-Yates, Ribeiro-Neto - 1999
568	Optimizing search engines using clickthrough data - Joachims - 2002
498	A study of smoothing methods for language models applied to information retrieval - Zhai, Lafferty
383	An efficient boosting algorithm for combining preferences - Freund, Iyer, et al. - 1998
349	UCI Machine Learning Repository - Asuncion, Newman
312	RCV1: A New Benchmark Collection for Text Categorization Research - Lewis, Yang, et al. - 2004
252	Learning to rank using gradient descent - Burges, Shaked, et al. - 2005
233	Cumulative gain-based evaluation of IR techniques - Järvelin, Kekäläinen - 2002
220	Combating web spam with trustrank - Gyöngyi, Garcia-Molina, et al.
204	D.M.: Mining the peanut gallery: opinion extraction and semantic classification of product reviews - Dave, Lawrence, et al. - 2003
198	OHSUMED: an interactive retrieval evaluation and new large test collection for research - Hersh, Buckley, et al. - 1994
115	Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales - Pang, Lee - 2005
106	Simple bm25 extension to multiple weighted fields - Robertson, Zaragoza, et al. - 2004
93	Beyond independent relevance: methods and evaluation metrics for subtopic retrieval - Zhai, Cohen, et al. - 2003
88	Learning to rank: From pairwise approach to listwise - Cao, Qin, et al. - 2007
76	Joachims T: A Support Vector Method for Optimizing Average Precision - Yue, Finley, et al. - 2007
67	TREC: Experiment and Evaluation in Information Retrieval - Voorhees - 2005
66	The TREC-9 filtering track final report - Robertson, Hull - 2001
61	Adapting ranking svm to document retrieval - Cao, Xu, et al. - 2006
60	Ranking algorithms for named-entity extraction: Boosting and the voted perceptron - Collins - 2002
60	Adarank: a boosting algorithm for information retrieval - Xu, Li - 2007
55	Support vector learning for ordinal regression - Herbrich, Graepel, et al. - 1999
49	Overview of the TREC2004 Web track - Craswell, Hawking - 2005
46	Overview of the TREC 2003 Web track - Craswell, Hawking - 2003
34	A study of relevance propagation for web search - Qin, Liu, et al. - 2005
31	Listwise approach to learning to rank: theory and algorithm - Xia, Liu, et al. - 2008
29	McRank: Learning to rank using multiple classification and gradient boosting - Li, Burges, et al. - 2008
26	Topical link analysis for web search - Nie, Davison, et al. - 2006
26	Frank: a ranking method with fidelity loss - Tsai, Liu, et al. - 2007
25	Softrank: optimizing non-smooth rank metrics - Taylor, Guiver, et al. - 2008
22	Online ranking/collaborative filtering using the perceptron algorithm - Harrington - 2003
20	L.: High accuracy retrieval with multiple nested ranker - Matveeva, Burges, et al. - 2006
20	Exploiting the hierarchical structure for link analysis - Xue, Yang, et al. - 2005
17	MailRank: using ranking for spam detection - Chirita, Diederich, et al. - 2005
17	Query dependent ranking using k-nearest neighbor - Geng, Liu, et al. - 2008
17	Ranking Definitions with Supervised Learning Methods - Xu, Cao, et al. - 2005
15	Query-level loss functions for information retrieval. Information processing and management - Qin, Zhang, et al. - 2007
15	Ranking with multiple hyperplanes - Qin, Zhang, et al. - 2007
13	Title Extraction from Bodies of HTML Documents and its Application to Web Page Retrieval - Hu
13	Ordinal Regression by Extended Binary Classification - Li, Lin - 2007
12	Learning to rank relational objects and its application to web search - Qin, Liu, et al. - 2008
12	Relevance Propagation for Topic Distillation UIUC TREC-2003 Web Track Experiments - Shakery, Zhai - 2003
11	Global ranking using continuous conditional random fields - Qin, Liu, et al. - 2008
11	Directly optimizing evaluation measures in learning to rank - Xu, Liu, et al. - 2008
8	Boltzrank: learning to maximize expected ranking gain - Volkovs, Zemel - 2009
6	Structured ranking learning using cumulative distribution networks - Huang, Frey - 2008
6	How to make letor more useful and reliable - Qin, Liu, et al. - 2008
5	Max margin classification of data with absent features - Chechik, Heitz, et al.
5	Selection bias in the letor datasets - Minka, Robertson - 2008