Directly optimizing evaluation measures in learning to rank

DMCA

Directly optimizing evaluation measures in learning to rank (2008)

Cached

Download Links

Other Repositories/Bibliography

DBLP

by Jun Xu , Tie-yan Liu , Min Lu , Hang Li , Wei-ying Ma

Venue:	In SIGIR ’08: Proceedings of the 31th annual international ACM SIGIR conference on Research and development in information retrieval
Citations:	33 - 8 self

BibTeX

@INPROCEEDINGS{Xu08directlyoptimizing,
    author = {Jun Xu and Tie-yan Liu and Min Lu and Hang Li and Wei-ying Ma},
    title = {Directly optimizing evaluation measures in learning to rank},
    booktitle = {In SIGIR ’08: Proceedings of the 31th annual international ACM SIGIR conference on Research and development in information retrieval},
    year = {2008},
    publisher = {ACM Press}
}

OpenURL

Abstract

One of the central issues in learning to rank for information retrieval is to develop algorithms that construct ranking models by directly optimizing evaluation measures used in information retrieval such as Mean Average Precision (MAP) and Normalized Discounted Cumulative Gain (NDCG). Several such algorithms including SVM map and AdaRank have been proposed and their effectiveness has been verified. However, the relationships between the algorithms are not clear, and furthermore no comparisons have been conducted between them. In this paper, we conduct a study on the approach of directly optimizing evaluation measures in learning to rank for Information Retrieval (IR). We focus on the methods that minimize loss functions upper bounding the basic loss function defined on the IR measures. We first provide a general framework for the study and analyze the existing algorithms of SVM map and AdaRank within the framework. The framework is based on upper bound analysis and two types of upper bounds are discussed. Moreover, we show that we can derive new algorithms on the basis of this analysis and create one example algorithm called PermuRank. We have also conducted comparisons between SVM map, AdaRank, PermuRank, and conventional methods of Ranking SVM and Rank-Boost, using benchmark datasets. Experimental results show that the methods based on direct optimization of evaluation measures can always outperform conventional methods of Ranking SVM and RankBoost. However, no significant difference exists among the performances of the direct optimization methods themselves.

Keyphrases

evaluation measure information retrieval svm map ranking svm conventional method significant difference direct optimization loss function central issue normalized discounted cumulative gain basic loss function ir measure upper bound analysis general framework example algorithm direct optimization method benchmark datasets several algorithm mean average precision new algorithm upper bound experimental result

DMCA