Results 1 - 10
of
35
LETOR: A Benchmark Collection for Research on Learning to Rank for Information Retrieval
"... LETOR is a benchmark collection for the research on learning to rank for information retrieval, released by Microsoft Research Asia. In this paper, we describe the details of the LETOR collection and show how it can be used in different kinds of researches. Specifically, we describe how the documen ..."
Abstract
-
Cited by 46 (3 self)
- Add to MetaCart
LETOR is a benchmark collection for the research on learning to rank for information retrieval, released by Microsoft Research Asia. In this paper, we describe the details of the LETOR collection and show how it can be used in different kinds of researches. Specifically, we describe how the document corpora and query sets in LETOR are selected, how the documents are sampled, how the learning features and meta information are extracted, and how the datasets are partitioned for comprehensive evaluation. We then compare several state-of-the-art learning to rank algorithms on LETOR, report their ranking performances, and make discussions on the results. After that, we discuss possible new research topics that can be supported by LETOR, in addition to algorithm comparison. We hope that this paper can help people to gain deeper understanding of LETOR, and enable more interesting research projects on learning to rank and related topics.
Global Ranking by Exploiting User Clicks
"... It is now widely recognized that user interactions with search results can provide substantial relevance information on the documents displayed in the search results. In this paper, we focus on extracting relevance information from one source of user interactions, i.e., user click data, which record ..."
Abstract
-
Cited by 25 (8 self)
- Add to MetaCart
(Show Context)
It is now widely recognized that user interactions with search results can provide substantial relevance information on the documents displayed in the search results. In this paper, we focus on extracting relevance information from one source of user interactions, i.e., user click data, which records the sequence of documents being clicked and not clicked in the result set during a user search session. We formulate the problem as a global ranking problem, emphasizing the importance of the sequential nature of user clicks, with the goal to predict the relevance labels of all the documents in a search session. This is distinct from conventional learning to rank methods that usually design a ranking model defined on a single document; in contrast, in our model the relational information among the documents as manifested by an aggregation of user clicks is exploited to rank all the documents jointly. In particular, we adapt several sequential supervised learning algorithms, including the conditional random field (CRF), the sliding window method and the recurrent sliding window method, to the global ranking problem. Experiments on the click data collected from a commercial search engine demonstrate that our methods can outperform the baseline models for search results re-ranking.
Improving verbose queries using subset distribution
- In Proc. CIKM
, 2010
"... Dealing with verbose (or long) queries poses a new challenge for information retrieval. Selecting a subset of the original query (a “sub-query”) has been shown to be an effective method for improving these queries. In this paper, the distribution of sub-queries (“subset distribution”) is formally mo ..."
Abstract
-
Cited by 18 (4 self)
- Add to MetaCart
(Show Context)
Dealing with verbose (or long) queries poses a new challenge for information retrieval. Selecting a subset of the original query (a “sub-query”) has been shown to be an effective method for improving these queries. In this paper, the distribution of sub-queries (“subset distribution”) is formally modeled within a well-grounded framework. Specifically, sub-query selection is considered as a sequential labeling problem, where each query word in a verbose query is assigned a label of “keep ” or “don’t keep”. A novel Conditional Random Field model is proposed to generate the distribution of sub-queries. This model captures the local and global dependencies between query words and directly optimizes the expected retrieval performance on a training set. The experiments, based on different retrieval models and performance measures, show that the proposed model can generate high-quality sub-query distributions and can significantly outperform state-of-the-art techniques.
Future directions in learning to rank
, 2011
"... The results of the learning to rank challenge showed that the quality of the predictions from the top competitors are very close from each other. This raises a question: is learning to rank a solved problem? On the on hand, it is likely that only small incremental progress can be made in the “core” ..."
Abstract
-
Cited by 16 (1 self)
- Add to MetaCart
The results of the learning to rank challenge showed that the quality of the predictions from the top competitors are very close from each other. This raises a question: is learning to rank a solved problem? On the on hand, it is likely that only small incremental progress can be made in the “core” and traditional problematics of learning to rank. The challenge was set in this standard learning to rank scenario: optimize a ranking measure on a test set. But on the other hand, there are a lot of related questions and settings in learning to rank that have not been yet fully explored. We review some of them in this paper and hope that researchers interested in learning to rank will try to answer these challenging and exciting research questions. 1. Learning Theory for Ranking Many learning to rank algorithms have been shown effective through benchmark experiments. However, sometimes benchmark experiments are not as reliable as expected due to the small scales of the training and test data. In this situation, a theory is needed to guarantee the performance of an algorithm on infinite unseen data.
Continuous Conditional Random Fields for Regression
- in Remote Sensing, Proc. 19 th European Conf. on Artificial Intelligence
"... Abstract. Conditional random fields (CRF) are widely used for predicting output variables that have some internal structure. Most of the CRF research has been done on structured classification where the outputs are discrete. In this study we propose a CRF probabilistic model for structured regressio ..."
Abstract
-
Cited by 14 (7 self)
- Add to MetaCart
(Show Context)
Abstract. Conditional random fields (CRF) are widely used for predicting output variables that have some internal structure. Most of the CRF research has been done on structured classification where the outputs are discrete. In this study we propose a CRF probabilistic model for structured regression that uses multiple non-structured predictors as its features. We construct features as squared prediction errors and show that this results in a Gaussian predictor. Learning becomes a convex optimization problem leading to a global solution for a set of parameters. Inference can be conveniently conducted through matrix computation. Experimental results on the remote sensing problem of estimating Aerosol Optical Depth (AOD) provide strong evidence that the proposed CRF model successfully exploits the inherent spatio-temporal properties of AOD data. The experiments revealed that CRF are more accurate than the baseline neural network and domain-based predictors. 1
Relational click prediction for sponsored search. WSDM,
, 2012
"... ABSTRACT This paper is concerned with the prediction of clicking an ad in sponsored search. The accurate prediction of user's click on an ad plays an important role in sponsored search, because it is widely used in both ranking and pricing of the ads. Previous work on click prediction usually ..."
Abstract
-
Cited by 13 (2 self)
- Add to MetaCart
(Show Context)
ABSTRACT This paper is concerned with the prediction of clicking an ad in sponsored search. The accurate prediction of user's click on an ad plays an important role in sponsored search, because it is widely used in both ranking and pricing of the ads. Previous work on click prediction usually takes a single ad as input, and ignores its relationship to the other ads shown in the same page. This independence assumption here, however, might not be valid in the real scenario. In this paper, we first perform an analysis on this issue by looking at the click-through rates (CTR) of the same ad, in the same position and for the same query, but surrounded by different ads. We found that in most cases the CTR varies largely, which suggests that the relationship between ads is really an important factor in predicting click probability. Furthermore, our investigation shows that the more similar the surrounding ads are to an ad, the lower the CTR of the ad is. Based on this observation, we design a continuous conditional random fields (CRF) based model for click prediction, which considers both the features of an ad and its similarity to the surrounding ads. We show that the model can be effectively learned using maximum likelihood estimation, and can also be efficiently inferred due to its closed form solution. Our experimental results on the click-through log from a commercial search engine show that the proposed model can predict clicks more accurately than previous independent models. To our best knowledge this is the first work that predicts ad clicks by considering the relationship between ads.
A social recommendation framework based on multi-scale continuous conditional random fields
- In Proc. of CIKM ’09
, 2009
"... This paper addresses the issue of social recommendation based on collaborative filtering (CF) algorithms. Social rec-ommendation emphasizes utilizing various attributes infor-mation and relations in social networks to assist recom-mender systems. Although recommendation techniques have obtained dist ..."
Abstract
-
Cited by 12 (2 self)
- Add to MetaCart
(Show Context)
This paper addresses the issue of social recommendation based on collaborative filtering (CF) algorithms. Social rec-ommendation emphasizes utilizing various attributes infor-mation and relations in social networks to assist recom-mender systems. Although recommendation techniques have obtained distinct developments over the decades, traditional CF algorithms still have these following two limitations: (1) relational dependency within predictions, an important fac-tor especially when the data is sparse, is not being uti-lized effectively; and (2) straightforward methods for com-bining features like linear integration suffer from high com-puting complexity in learning the weights by enumerating the whole value space, making it difficult to combine var-ious information into an unified approach. In this paper, we propose a novel model, Multi-scale Continuous Condi-tional Random Fields (MCCRF), as a framework to solve above problems for social recommendations. In MCCRF, relational dependency within predictions is modeled by the Markov property, thus predictions are generated simultane-ously and can help each other. This strategy has never been employed previously. Besides, diverse information and rela-tions in social network can be modeled by state and edge feature functions in MCCRF, whose weights can be opti-mized globally. Thus both problems can be solved under this framework. In addition, We propose to utilize Markov chain Monte Carlo (MCMC) estimation methods to solve the difficulties in training and inference processes of MCCRF. Experimental results conducted on two real world data have demonstrated that our approach outperforms traditional CF algorithms. Additional experiments also show the improve-ments from the two factors of relational dependency and feature combination, respectively.
Actionness ranking with lattice conditional ordinal random fields
- In CVPR
, 2014
"... Action analysis in image and video has been attracting more and more attention in computer vision. Recognizing specific actions in video clips has been the main focus. We move in a new, more general direction in this paper and ask the critical fundamental question: what is action, how is action diff ..."
Abstract
-
Cited by 8 (4 self)
- Add to MetaCart
(Show Context)
Action analysis in image and video has been attracting more and more attention in computer vision. Recognizing specific actions in video clips has been the main focus. We move in a new, more general direction in this paper and ask the critical fundamental question: what is action, how is action different from motion, and in a given image or video where is the action? We study the philosophical and vi-sual characteristics of action, which lead us to define ac-tionness: intentional bodily movement of biological agents (people, animals). To solve the general problem, we pro-pose the lattice conditional ordinal random field model that incorporates local evidence as well as neighboring order agreement. We implement the new model in the continuous domain and apply it to scoring actionness in both image and video datasets. Our experiments demonstrate not only that our new model can outperform the popular ranking SVM but also that indeed action is distinct from motion. 1.
Exponential family graph matching and ranking
- CoRR
"... We present a method for learning max-weight matching predictors in bipartite graphs. The method consists of performing maximum a posteriori estimation in exponential families with sufficient statistics that encode permutations and data features. Although inference is in general hard, we show that fo ..."
Abstract
-
Cited by 7 (0 self)
- Add to MetaCart
(Show Context)
We present a method for learning max-weight matching predictors in bipartite graphs. The method consists of performing maximum a posteriori estimation in exponential families with sufficient statistics that encode permutations and data features. Although inference is in general hard, we show that for one very relevant application–document ranking–exact inference is efficient. For general model instances, an appropriate sampler is readily available. Contrary to existing max-margin matching models, our approach is statistically consistent and, in addition, experiments with increasing sample sizes indicate superior improvement over such models. We apply the method to graph matching in computer vision as well as to a standard benchmark dataset for learning document ranking, in which we obtain state-of-the-art results, in particular improving on max-margin variants. The drawback of this method with respect to max-margin alternatives is its runtime for large graphs, which is comparatively high. 1
Continuous conditional random fields for efficient regression in large fully connected graphs
, 2013
"... When used for structured regression, powerful Condi-tional Random Fields (CRFs) are typically restricted to modeling effects of interactions among examples in lo-cal neighborhoods. Using more expressive representa-tion would result in dense graphs, making these meth-ods impractical for large-scale a ..."
Abstract
-
Cited by 6 (2 self)
- Add to MetaCart
(Show Context)
When used for structured regression, powerful Condi-tional Random Fields (CRFs) are typically restricted to modeling effects of interactions among examples in lo-cal neighborhoods. Using more expressive representa-tion would result in dense graphs, making these meth-ods impractical for large-scale applications. To address this issue, we propose an effective CRF model with linear scale-up properties regarding approximate learn-ing and inference for structured regression on large, fully connected graphs. The proposed method is vali-dated on real-world large-scale problems of image de-noising and remote sensing. In conducted experiments, we demonstrated that dense connectivity provides an improvement in prediction accuracy. Inference time of less than ten seconds on graphs with millions of nodes and trillions of edges makes the proposed model an at-tractive tool for large-scale, structured regression prob-lems.
