Results 1  10
of
29
Optimistic Knowledge Gradient Policy for Optimal Budget Allocation in Crowdsourcing
"... We consider the budget allocation problem in binary/multiclass crowd labeling where each label from the crowd has a certain cost. Since different instances have different ambiguities and different workers have different reliabilities, a fundamental challenge here is how to allocate a prefixed amou ..."
Abstract

Cited by 19 (3 self)
 Add to MetaCart
(Show Context)
We consider the budget allocation problem in binary/multiclass crowd labeling where each label from the crowd has a certain cost. Since different instances have different ambiguities and different workers have different reliabilities, a fundamental challenge here is how to allocate a prefixed amount of budget among instanceworker pairs so that the overall accuracy can be maximized. We start with a simple setting where all workers are assumed to be identical and formulate the problem as a Bayesian Markov Decision Process (MDP). Using the dynamic programming (DP) algorithm, one can obtain the optimal allocation policy for any given budget. However, DP is computationally intractable. To address the computational challenge, we propose a new approximate policy, called optimistic knowledge gradient. The consistency of the proposed policy is established. Then we extend the MDP framework to incorporate estimating the reliabilities of workers into the allocation process when workers are no longer identical. We conduct simulated and real experiments to demonstrate the superiority of our policy in different crowd labeling tasks.
Spectral methods meet EM: A provably optimal algorithm for crowdsourcing. In Advances in neural information processing systems,
, 2014
"... Abstract Crowdsourcing is a popular paradigm for effectively collecting labels at low cost. The DawidSkene estimator has been widely used for inferring the true labels from the noisy labels provided by nonexpert crowdsourcing workers. However, since the estimator maximizes a nonconvex loglikeli ..."
Abstract

Cited by 15 (2 self)
 Add to MetaCart
(Show Context)
Abstract Crowdsourcing is a popular paradigm for effectively collecting labels at low cost. The DawidSkene estimator has been widely used for inferring the true labels from the noisy labels provided by nonexpert crowdsourcing workers. However, since the estimator maximizes a nonconvex loglikelihood function, it is hard to theoretically justify its performance. In this paper, we propose a twostage efficient algorithm for multiclass crowd labeling problems. The first stage uses the spectral method to obtain an initial estimate of parameters. Then the second stage refines the estimation by optimizing the objective function of the DawidSkene estimator via the EM algorithm. We show that our algorithm achieves the optimal convergence rate up to a logarithmic factor. We conduct extensive experiments on synthetic and real datasets. Experimental results demonstrate that the proposed algorithm is comparable to the most accurate empirical approach, while outperforming several other recently proposed methods.
Aggregating crowdsourced binary ratings
, 2013
"... In this paper we analyze a crowdsourcing system consisting of a set of users and a set of binary choice questions. Each user has an unknown, fixed, reliability that determines the user’s error rate in answering questions. The problem is to determine the truth values of the questions solely based on ..."
Abstract

Cited by 15 (1 self)
 Add to MetaCart
In this paper we analyze a crowdsourcing system consisting of a set of users and a set of binary choice questions. Each user has an unknown, fixed, reliability that determines the user’s error rate in answering questions. The problem is to determine the truth values of the questions solely based on the user answers. Although this problem has been studied extensively, theoretical error bounds have been shown only for restricted settings: when the graph between users and questions is either random or complete. In this paper we consider a general setting of the problem where the user–question graph can be arbitrary. We obtain bounds on the error rate of our algorithm and show it is governed by the expansion of the graph. We demonstrate, using several synthetic and real datasets, that our algorithm outperforms the state of the art.
A ConfidenceAware Approach for Truth Discovery on LongTail Data
"... In many real world applications, the same item may be described by multiple sources. As a consequence, conflicts among these sources are inevitable, which leads to an important task: how to identify which piece of information is trustworthy, i.e., the truth discovery task. Intuitively, if the piece ..."
Abstract

Cited by 11 (8 self)
 Add to MetaCart
(Show Context)
In many real world applications, the same item may be described by multiple sources. As a consequence, conflicts among these sources are inevitable, which leads to an important task: how to identify which piece of information is trustworthy, i.e., the truth discovery task. Intuitively, if the piece of information is from a reliable source, then it is more trustworthy, and the source that provides trustworthy information is more reliable. Based on this principle, truth discovery approaches have been proposed to infer source reliability degrees and the most trustworthy information (i.e., the truth) simultaneously. However, existing approaches overlook the ubiquitous longtail phenomenon in the tasks, i.e., most sources only provide a few claims and only a few sources make plenty of claims, which causes the source reliability estimation for small sources to be unreasonable. To tackle this challenge, we propose a confidenceaware truth discovery (CATD) method to automatically detect truths from conflicting data with longtail phenomenon. The proposed method not only estimates source reliability, but also considers the confidence interval of the estimation, so that it can effectively reflect real source reliability for sources with various levels of participation. Experiments on four real world tasks as well as simulated multisource longtail datasets demonstrate that the proposed method outperforms existing stateoftheart truth discovery approaches by successful discounting the effect of small sources. 1.
Communitybased bayesian aggregation models for crowdsourcing.
 In WWW,
, 2014
"... ABSTRACT This paper addresses the problem of extracting accurate labels from crowdsourced datasets, a key challenge in crowdsourcing. Prior work has focused on modeling the reliability of individual workers, for instance, by way of confusion matrices, and using these latent traits to estimate the t ..."
Abstract

Cited by 11 (1 self)
 Add to MetaCart
(Show Context)
ABSTRACT This paper addresses the problem of extracting accurate labels from crowdsourced datasets, a key challenge in crowdsourcing. Prior work has focused on modeling the reliability of individual workers, for instance, by way of confusion matrices, and using these latent traits to estimate the true labels more accurately. However, this strategy becomes ineffective when there are too few labels per worker to reliably estimate their quality. To mitigate this issue, we propose a novel communitybased Bayesian label aggregation model, CommunityBCC, which assumes that crowd workers conform to a few different types, where each type represents a group of workers with similar confusion matrices. We assume that each worker belongs to a certain community, where the worker's confusion matrix is similar to (a perturbation of) the community's confusion matrix. Our model can then learn a set of key latent features: (i) the confusion matrix of each community, (ii) the community membership of each user, and (iii) the aggregated label of each item. We compare the performance of our model against established aggregation methods on a number of largescale, realworld crowdsourcing datasets. Our experimental results show that our CommunityBCC model consistently outperforms stateoftheart label aggregation methods, gaining, on average, 8% more accuracy with the same amount of labels.
Inferring Users ’ Preferences from Crowdsourced Pairwise Comparisons: A Matrix Completion Approach
"... Inferring user preferences over a set of items is an important problem that has found numerous applications. This work focuses on the scenario where the explicit feature representation of items is unavailable, a setup that is similar to collaborative filtering. In order to learn a user’s preference ..."
Abstract

Cited by 7 (0 self)
 Add to MetaCart
Inferring user preferences over a set of items is an important problem that has found numerous applications. This work focuses on the scenario where the explicit feature representation of items is unavailable, a setup that is similar to collaborative filtering. In order to learn a user’s preferences from his/her response to only a small number of pairwise comparisons, we propose to leverage the pairwise comparisons made by many crowd users, a problem we refer to as crowdranking. The proposed crowdranking framework is based on the theory of matrix completion, and we present efficient algorithms for solving the related optimization problem. Our theoretical analysis shows that, on average, only O(r logm) pairwise queries are needed to accurately recover the ranking list of m items for the target user, where r is the rank of the unknown rating matrix, r m. Our empirical study with two realworld benchmark datasets for collaborative filtering and one crowdranking dataset we collected via Amazon Mechanical Turk shows the promising performance of the proposed algorithm compared to the stateoftheart approaches.
Scoring Workers in Crowdsourcing: How Many Control Questions are Enough?
"... We study the problem of estimating continuous quantities, such as prices, probabilities, and point spreads, using a crowdsourcing approach. A challenging aspect of combining the crowd’s answers is that workers ’ reliabilities and biases are usually unknown and highly diverse. Control items with know ..."
Abstract

Cited by 6 (3 self)
 Add to MetaCart
(Show Context)
We study the problem of estimating continuous quantities, such as prices, probabilities, and point spreads, using a crowdsourcing approach. A challenging aspect of combining the crowd’s answers is that workers ’ reliabilities and biases are usually unknown and highly diverse. Control items with known answers can be used to evaluate workers ’ performance, and hence improve the combined results on the target items with unknown answers. This raises the problem of how many control items to use when the total number of items each workers can answer is limited: more control items evaluates the workers better, but leaves fewer resources for the target items that are of direct interest, and vice versa. We give theoretical results for this problem under different scenarios, and provide a simple rule of thumb for crowdsourcing practitioners. As a byproduct, we also provide theoretical analysis of the accuracy of different consensus methods. 1
Big Data Opportunities and Challenges: Discussions from Data Analytics Perspectives
"... Abstract—“Big Data ” as a term has been among the biggest trends of the last three years, leading to an upsurge of research, as well as industry and government applications. Data is deemed a powerful raw material that can impact multidisciplinary research endeavors as well as government and business ..."
Abstract

Cited by 3 (0 self)
 Add to MetaCart
(Show Context)
Abstract—“Big Data ” as a term has been among the biggest trends of the last three years, leading to an upsurge of research, as well as industry and government applications. Data is deemed a powerful raw material that can impact multidisciplinary research endeavors as well as government and business performance. The goal of this discussion paper is to share the data analytics opinions and perspectives of the authors relating to the new opportunities and challenges brought forth by the big data movement. The authors bring together diverse perspectives, coming from different geographical locations with different core research expertise and different affiliations and work experiences. The aim of this paper is to evoke discussion rather than to provide a comprehensive survey of big data research. Index Terms—Big data, data analytics, machine learning, data mining, global optimization, application F 1
MaxMargin Majority Voting for Learning from Crowds
"... Abstract Learningfromcrowds aims to design proper aggregation strategies to infer the unknown true labels from the noisy labels provided by ordinary web workers. This paper presents maxmargin majority voting (M 3 V) to improve the discriminative ability of majority voting and further presents a ..."
Abstract

Cited by 1 (0 self)
 Add to MetaCart
(Show Context)
Abstract Learningfromcrowds aims to design proper aggregation strategies to infer the unknown true labels from the noisy labels provided by ordinary web workers. This paper presents maxmargin majority voting (M 3 V) to improve the discriminative ability of majority voting and further presents a Bayesian generalization to incorporate the flexibility of generative methods on modeling noisy observations with worker confusion matrices. We formulate the joint learning as a regularized Bayesian inference problem, where the posterior regularization is derived by maximizing the margin between the aggregated score of a potential true label and that of any alternative label. Our Bayesian model naturally covers the DawidSkene estimator and M 3 V. Empirical results demonstrate that our methods are competitive, often achieving better results than stateoftheart estimators.
Approval voting and incentives in crowdsourcing
 In Proc. of 32nd ICML
, 2015
"... Abstract The growing need for labeled training data has made crowdsourcing an important part of machine learning. The quality of crowdsourced labels is, however, adversely affected by three factors: (1) the workers are not experts; (2) the incentives of the workers are not aligned with those of the ..."
Abstract

Cited by 1 (1 self)
 Add to MetaCart
(Show Context)
Abstract The growing need for labeled training data has made crowdsourcing an important part of machine learning. The quality of crowdsourced labels is, however, adversely affected by three factors: (1) the workers are not experts; (2) the incentives of the workers are not aligned with those of the requesters; and (3) the interface does not allow workers to convey their knowledge accurately, by forcing them to make a single choice among a set of options. In this paper, we address these issues by introducing approval voting to utilize the expertise of workers who have partial knowledge of the true answer, and coupling it with a ("strictly proper") incentivecompatible compensation mechanism. We show rigorous theoretical guarantees of optimality of our mechanism together with a simple axiomatic characterization. We also conduct preliminary empirical studies on Amazon Mechanical Turk which validate our approach.