Results 1 -
9 of
9
Dwelling on the Negative: Incentivizing Effort in Peer Prediction
"... Agents are asked to rank two objects in a setting where effort is costly and agents differ in quality (which is the probability that they can identify the correct, ground truth, ranking). We study simple output-agreement mechanisms that pay an agent in the case she agrees with the report of another, ..."
Abstract
-
Cited by 6 (0 self)
- Add to MetaCart
Agents are asked to rank two objects in a setting where effort is costly and agents differ in quality (which is the probability that they can identify the correct, ground truth, ranking). We study simple output-agreement mechanisms that pay an agent in the case she agrees with the report of another, and potentially penalizes for disagreement through a negative payment. Assuming access to a quality oracle, able to determine whether an agent’s quality is above a given threshold, we design a payment scheme that aligns incentives so that agents whose quality is above this threshold participate and invest effort. Precluding negative payments leads the expected cost of this quality-oracle mechanism to increase by a factor of 2 to 5 relative to allowing both positive and negative payments. Dropping the assumption about access to a quality oracle, we further show that negative payments can be used to make agents with quality lower than the quality threshold choose to not to participate, while those above continue to participate and invest effort. Through the appropriate choice of payments, any design threshold can be achieved. This selfselection mechanism has the same expected cost as the costminimal quality-oracle mechanism, and thus when using the self-selection mechanism, perfect screening comes for free.
Trick or Treat: Putting Peer Prediction to the Test
"... Collecting subjective information from multiple parties is a common problem in collective intelligence. However, incentivizing truthful reports is difficult when there is no ground truth to verify the reports against. Peer prediction mechanisms use collected reports alone to provide good theoretical ..."
Abstract
-
Cited by 2 (0 self)
- Add to MetaCart
(Show Context)
Collecting subjective information from multiple parties is a common problem in collective intelligence. However, incentivizing truthful reports is difficult when there is no ground truth to verify the reports against. Peer prediction mechanisms use collected reports alone to provide good theoretical incentives for truthful reporting, but make assumptions that are difficult to satisfy in the real world. They also admit uninformative equilibria where coordinating participants provide no useful information. Using a multiplayer, real-time repeated game, we conduct the first controlled online experiment of a peer prediction method. Our results show that players learn to adopt more profitable strategies through repeated use of the mechanism, and that there is a distinct incentive for participants to converge to the uninformative equilibria. 1
Approval voting and incentives in crowdsourcing
- In Proc. of 32nd ICML
, 2015
"... Abstract The growing need for labeled training data has made crowdsourcing an important part of machine learning. The quality of crowdsourced labels is, however, adversely affected by three factors: (1) the workers are not experts; (2) the incentives of the workers are not aligned with those of the ..."
Abstract
-
Cited by 1 (1 self)
- Add to MetaCart
(Show Context)
Abstract The growing need for labeled training data has made crowdsourcing an important part of machine learning. The quality of crowdsourced labels is, however, adversely affected by three factors: (1) the workers are not experts; (2) the incentives of the workers are not aligned with those of the requesters; and (3) the interface does not allow workers to convey their knowledge accurately, by forcing them to make a single choice among a set of options. In this paper, we address these issues by introducing approval voting to utilize the expertise of workers who have partial knowledge of the true answer, and coupling it with a ("strictly proper") incentive-compatible compensation mechanism. We show rigorous theoretical guarantees of optimality of our mechanism together with a simple axiomatic characterization. We also conduct preliminary empirical studies on Amazon Mechanical Turk which validate our approach.
Incentives to Counter Bias in Human Computation
"... In online labor platforms such as Amazon Mechanical Turk, a good strategy to obtain quality answers is to take aggregate answers submitted by multiple workers, exploiting the wis-dom of the crowd. However, human computation is suscep-tible to systematic biases which cannot be corrected by us-ing mul ..."
Abstract
-
Cited by 1 (0 self)
- Add to MetaCart
In online labor platforms such as Amazon Mechanical Turk, a good strategy to obtain quality answers is to take aggregate answers submitted by multiple workers, exploiting the wis-dom of the crowd. However, human computation is suscep-tible to systematic biases which cannot be corrected by us-ing multiple workers. We investigate a game-theoretic bonus scheme, called Peer Truth Serum (PTS), to overcome this problem. We report on the design and outcomes of a set of experiments to validate this scheme. Results show Peer Truth Serum can indeed correct the biases and increase the answer accuracy by up to 80%.
Elicitability and Knowledge-Free Elicitation with Peer Prediction
"... The elicitation of private information from individuals is crucially important to many real-world tasks. But elicitation is most challenging when it is most useful: when objective (verifiable) truth is inaccessible or unavailable, and there is no “answer key ” available to verify reports. Prior work ..."
Abstract
-
Cited by 1 (0 self)
- Add to MetaCart
(Show Context)
The elicitation of private information from individuals is crucially important to many real-world tasks. But elicitation is most challenging when it is most useful: when objective (verifiable) truth is inaccessible or unavailable, and there is no “answer key ” available to verify reports. Prior work has designed mechanisms that truthfully elicit private information without verification for some restricted set of possible information structures of the participants (i.e. the common prior joint distributions of participants ’ signals). In fact, no mechanism can elicit private information truthfully for all information structures without verification. In this paper, we identify the maximal set of information structures that are truthfully elicitable without verification, and provide a mechanism for such elicitation. This mechanism requires that the designer know the information structure of the participants, which is unavailable in many settings. We then propose a knowledge-free peer prediction mechanism that does not require knowledge of the information structure and can truthfully elicit private information for a set of information structures slightly smaller than the maximal set. This mechanism works for both small and large populations in settings with both binary and non-binary private signals, and is effective on a strict superset of information structures as compared to prior mechanisms that satisfy these properties.
Tuned models of peer . . .
, 2013
"... In massive open-access online courses (MOOCs), peer grading serves as a critical tool for scaling the grading of complex, open-ended assignments to courses with tens or hundreds of thousands of students. But despite promising initial trials, it does not always deliver accurate results compared to hu ..."
Abstract
- Add to MetaCart
In massive open-access online courses (MOOCs), peer grading serves as a critical tool for scaling the grading of complex, open-ended assignments to courses with tens or hundreds of thousands of students. But despite promising initial trials, it does not always deliver accurate results compared to human experts. In this paper, we develop algorithms for estimating and correcting for grader biases and reliabilities, showing significant improvement in peer grading accuracy on real data with 63,199 peer grades from Coursera’s HCI course offerings — the largest peer grading networks analysed to date. We relate grader biases and reliabilities to other student factors such as engagement, performance as well as commenting style. We also show that our model can lead to more intelligent assignment of graders to gradees.
Output Agreement Mechanisms and Common Knowledge
"... The recent advent of human computation – employing non-experts to solve problems – has inspired theoretical work in mechanism design for eliciting information when responses cannot be verified. We study a popular practical method, output agreement, from a theoretical perspective. In output agreement ..."
Abstract
- Add to MetaCart
The recent advent of human computation – employing non-experts to solve problems – has inspired theoretical work in mechanism design for eliciting information when responses cannot be verified. We study a popular practical method, output agreement, from a theoretical perspective. In output agreement, two agents are given the same inputs and asked to produce some output; they are scored based on how closely their re-sponses agree. Although simple, output agreement raises new conceptual questions. Primary is the fundamental importance of common knowledge: We show that, rather than being truthful, output agreement mechanisms elicit common knowledge from par-ticipants. We show that common knowledge is essentially the best that can be hoped for in any mechanism without verifica-tion unless there are restrictions on the information structure. This involves generalizing truthfulness to include responding to a query rather than simply reporting a private signal, along with a notion of common-knowledge equilibria. A final impor-tant issue raised by output agreement is focal equilibria and player computation of equilibria. We show that, for eliciting the mean of a random variable, a natural player inference pro-cess converges to the common-knowledge equilibrium; but this convergence may not occur for other types of queries.
