Citation Context

... the expected risk of a learned function converges to the Bayes risk as the training sample size increases (Lin, 2002, Zhang, 2004b, Steinwart, 2005, Bartlett et al., 2006, Tewari and Bartlett, 2007, =-=Duchi et al., 2010-=-). Nowadays, it is well-accepted that a good learner should at least be consistent with large samples. It is noteworthy that, though many efforts have been devoted to multi-label learning, few theoret...

Citation Context

...y distribution ˜ P . This error metric provides us with a means to quantify the relative certainty in guessing if one item is preferred over another. Furthermore, producing such scores are ubiquitous =-=[17]-=- as they may also be used to calculate the desired rankings. After presenting our main theoretical result we will then provide simulations demonstrating the empirical performance of our algorithm in d...

Citation Context

... under finite samples? A line of recent results have studied consistency and Bayes optimality of estimates for the cases where the target evaluation measure is pointwise [10], and when it is pairwise =-=[9, 12]-=-, and for the zero-one listwise loss [6]. In this paper, we study the consistency of any surrogate ranking loss function with respect to the listwise NDCG evaluation measure. We first provide a charac...

Citation Context

... explore a stabilitybased analysis for this algorithm [7, 4]. Second, it would be of interest to understand consistency properties of the proposed algorithm; in general, with a few notable exceptions =-=[14, 16, 17]-=-, there is limited understanding of consistency for ranking algorithms. Finally, for simplicity, we have focused in this paper on the bipartite ranking setting; however it should be possible to extend...

Citation Context

...a to minimize the Ψ-risk: R Ψ (f) = E [ Ψ(f(Q), T) ] . (3) The goal of the paper is to give general conditions on Ψ and R s such that Ψ is consistent with respect to R s (see Def. 2 below). Th. 1 of (=-=Duchi et al., 2010-=-) states that 2 We will make an exception in Sections 5.3 and 5.4, where the supervisions considered are respectively weighted preference graphs and permutations.Order-Preserving Losses and Standardi...

Citation Context

...is an inconsistent learning procedure, which may converge to non-optimal predictions as the number of examples increases. Our result generalizes previous works on non-calibration. First, Duchi et al. =-=[11]-=- showed that many convex losses based on pairwise comparisons, such as those of RankBoost [12] or Ranking SVMs [14, 4], are not calibrated with respect to the PD. Secondly, Buffoni et al. [2] showed t...

Citation Context

... upper and lower bounds on this quantity, and use these results to analyze various loss matrices. As one application, in Section 5, we provide a different route from the recent result of Duchi et al. =-=[10]-=- for analyzing the difficulty of designing ‘low-dimensional’ convex surrogates that are consistent with respect to certain pairwise subset ranking losses. We conclude in Section 6 with some future dir...

Citation Context

...ly applied. A learning-to-rank process can be described as follows. In training, a number of sets (queries) of objects (documents) are given and within each set the objects are labeled by assessors, mainly based on multi-level ratings. The target of learning is to create a model that provides a ranking over the objects that best respects the observed labels. In testing, given a new set of objects, the trained model is applied to generate a ranked list of the objects. Ideally, the learning process should be guided by minimizing a true loss such as the weighted pairwise disagreement loss (WPDL) [11], which encodes people’s knowledge on ranking evaluation. However, the minimization can be very difficult due to the nonconvexity of the true loss. Alternatively, many learning-to-rank methods minimize surrogate loss functions. For example, RankSVM [14], RankBoost [12], and RankNet [3] minimize the hinge loss, the exponential loss, and the crossentropy loss, respectively. In machine learning, statistical consistency is regarded as a desired property of a learning method [1, 21, 20], which reveals the statistical connection between a surrogate loss function and the true loss. Statistical consis...

Citation Context

... sparse mixture model, i.e. distribution over permutations with each mixture being delta distribution, [11] provided sufficient conditions under which mixtures can be learnt exactly using pair-wise marginals – effectively, as long as the number of components scaled as o(log n) where components satisfied appropriate incoherence condition, a simple iterative algorithm could recover the mixture. However, it is not robust with respect to noise in data or finite sample error in marginal estimation. Other approaches have been proposed to recover model using convex optimization based techniques, cf. [10, 18]. MNL model is a special case of a larger family of discrete choice models known as the Random Utility Model (RUM), and an efficient algorithm to learn RUM is introduced in [22]. Efficient algorithms for learning RUMs from partial rankings has been introduced in [3, 4]. We note that the above list of references is very limited, including only closely related literature. Given the nature of the topic, there are a lot of exciting lines of research done over the past century and we shall not be able to provide comprehensive coverage due to a space limitation. Problem. Given observations from the ...