DMCA
Identifying and Accounting for Task-Dependent Bias in Crowdsourcing
BibTeX
@MISC{Kamar_identifyingand,
author = {Ece Kamar and Ashish Kapoor and Eric Horvitz},
title = {Identifying and Accounting for Task-Dependent Bias in Crowdsourcing},
year = {}
}
Share
 |
 |
 |
 |
||
OpenURL
Abstract
Models for aggregating contributions by crowd workers have been shown to be challenged by the rise of task-specific biases or errors. Task-dependent errors in as-sessment may shift the majority opinion of even large numbers of workers to an incorrect answer. We intro-duce and evaluate probabilistic models that can detect and correct task-dependent bias automatically. First, we show how to build and use probabilistic graphical mod-els for jointly modeling task features, workers ’ biases, worker contributions and ground truth answers of tasks so that task-dependent bias can be corrected. Second, we show how the approach can perform a type of trans-fer learning among workers to address the issue of an-notation sparsity. We evaluate the models with varying complexity on a large data set collected from a citizen science project and show that the models are effective at correcting the task-dependent worker bias. Finally, we investigate the use of active learning to guide the acquisition of expert assessments to enable automatic detection and correction of worker bias.
Keyphrases
task-dependent bias worker bias incorrect answer trans-fer learning large data set crowd worker probabilistic graphical mod-els probabilistic model expert assessment active learning task feature automatic detection citizen science project task-dependent error large number task-specific bias an-notation sparsity worker contribution task-dependent worker bias ground truth answer majority opinion
