Results 1 -
6 of
6
Toward a learning science for complex crowdsourcing tasks.
- In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems,
, 2016
"... ABSTRACT We explore how crowdworkers can be trained to tackle complex crowdsourcing tasks. We are particularly interested in training novice workers to perform well on solving tasks in situations where the space of strategies is large and workers need to discover and try different strategies to be ..."
Abstract
-
Cited by 2 (2 self)
- Add to MetaCart
(Show Context)
ABSTRACT We explore how crowdworkers can be trained to tackle complex crowdsourcing tasks. We are particularly interested in training novice workers to perform well on solving tasks in situations where the space of strategies is large and workers need to discover and try different strategies to be successful. In a first experiment, we perform a comparison of five different training strategies. For complex web search challenges, we show that providing expert examples is an effective form of training, surpassing other forms of training in nearly all measures of interest. However, such training relies on access to domain expertise, which may be expensive or lacking. Therefore, in a second experiment we study the feasibility of training workers in the absence of domain expertise. We show that having workers validate the work of their peer workers can be even more effective than having them review expert examples if we only present solutions filtered by a threshold length. The results suggest that crowdsourced solutions of peer workers may be harnessed in an automated training pipeline.
Combining crowdsourcing and learning to improve engagement and performance
- Proceedings of the 32nd annual ACM conference on Human factors in computing systems: ACM
, 2014
"... original improved Figure 1. LevelUp for Photoshop is a crowdsourcing platform that combines learning and creative work. Workers learn photo editing skills, while improving real-world images. Interactive step-by-step tutorials teach workers new techniques and Challenge Rounds filled with images from ..."
Abstract
-
Cited by 2 (0 self)
- Add to MetaCart
(Show Context)
original improved Figure 1. LevelUp for Photoshop is a crowdsourcing platform that combines learning and creative work. Workers learn photo editing skills, while improving real-world images. Interactive step-by-step tutorials teach workers new techniques and Challenge Rounds filled with images from different requester organizations test worker knowledge. The worker interface is shown on the left; several original and improved images are shown on the right. Crowdsourcing complex creative tasks remains difficult, in part because these tasks require skilled workers. Most crowd-sourcing platforms do not help workers acquire the skills nec essary to accomplish complex creative tasks. In this paper, we describe a platform that combines learning and crowdsourc ing to benefit both the workers and the requesters. Workers gain new skills through interactive step-by-step tutorials and test their knowledge by improving real-world images submit ted by requesters. In a series of three deployments spanning two years, we varied the design of our platform to enhance the learning experience and improve the quality of the crowd work. We tested our approach in the context of LevelUp for Photoshop, which teaches people how to do basic photograph improvement tasks using Adobe Photoshop. We found that by using our system workers gained new skills and produced high-quality edits for requested images, even if they had little prior experience editing images.
A System for Scalable and Reliable Technical-Skill Testing in Online Labor Markets
"... Abstract The emergence of online labor platforms, online crowdsourcing sites, and even Massive Open Online Courses (MOOCs), has created an increasing need for reliably evaluating the skills of the participating users (e.g., "does a candidate know Java") in a scalable way. Many platforms a ..."
Abstract
- Add to MetaCart
(Show Context)
Abstract The emergence of online labor platforms, online crowdsourcing sites, and even Massive Open Online Courses (MOOCs), has created an increasing need for reliably evaluating the skills of the participating users (e.g., "does a candidate know Java") in a scalable way. Many platforms already allow job candidates to take online tests to asses their competence in a variety of technical topics. However the existing approaches face many problems. First, cheating is very common in online testing without supervision, as the test questions often "leak" and become easily available online along with the answers. Second, technical-skills, such as programming, require the tests to be frequently updated in order to reflect the current state-of-the-art. Third, there is very limited evaluation of the tests themselves, and how effectively they measure the skill that the users are tested for. In this article we present a platform, that continuously generates test questions and evaluates their quality as predictors of the user skill level. Our platform leverages content that is already available on question answering sites such as Stack Overflow and re-purposes these questions to generate tests. This approach has some major benefits: we continuously generate new questions, decreasing the impact of cheating, and we also create questions that are closer to the real problems that the skill holder is expected to solve in real life. Our platform leverages the use of Item Response Theory to evaluate the quality of the questions. We also use external signals about the quality of the workers to examine the external validity of the generated test questins: Questions that have external validity also have a strong predictive ability for identifying early the workers that have the potential to succeed in the online job marketplaces. Our experimental evaluation shows that our system generates questions of comparable or higher quality compared to existing tests, with a cost of approximately $3 to $5 dollars per question, which is lower than the cost of licensing questions from existing test banks, and an order of magnitude lower than the cost of producing such questions from scratch using experts.
STEP: A Scalable Testing and Evaluation Platform
"... Abstract The emergence of online crowdsourcing sites, online work platforms, and even Massive Open Online Courses (MOOCs), has created an increasing need for reliably evaluating the skills of the participating users in a scalable way. Many platforms already allow users to take online tests and veri ..."
Abstract
- Add to MetaCart
(Show Context)
Abstract The emergence of online crowdsourcing sites, online work platforms, and even Massive Open Online Courses (MOOCs), has created an increasing need for reliably evaluating the skills of the participating users in a scalable way. Many platforms already allow users to take online tests and verify their skills, but the existing approaches face many problems. First of all, cheating is very common in online testing without supervision, as the test questions often "leak" and become easily available online together with the answers. Second, technical skills, such as programming, require the tests to be frequently updated in order to reflect the current state-of-the-art. Third, there is very limited evaluation of the tests themselves, and how effectively they measure the skill that the users are tested for. In this paper, we present a Scalable Testing and Evaluation Platform (STEP), that allows continuous generation and evaluation of test questions. STEP leverages already available content, on Question Answering sites such as Stack Overflow and re-purposes these questions to generate tests. The system utilizes a crowdsourcing component for the editing of the questions, while it uses automated techniques for identifying promising QA threads that can be successfully re-purposed for testing. This continuous question generation decreases the impact of cheating and also creates questions that are closer to the real problems that the skill holder is expected to solve in real life. STEP also leverages the use of Item Response Theory to evaluate the quality of the questions. We also use external signals about the quality of the workers. These identify the questions that have the strongest predictive ability in distinguishing workers that have the potential to succeed in the online job marketplaces. Existing approaches contrast in using only internal consistency metrics to evaluate the questions. Finally, our system employs an automatic "leakage detector" that queries the Internet to identify leaked versions of our questions. We then mark these questions as "practice only," effectively removing them from the pool of questions used for evaluation. Our experimental evaluation shows that our system generates questions of comparable or higher quality compared to existing tests, with a cost of approximately 3−5 dollars per question, which is lower than the cost of licensing questions from existing test banks.
Structuring Interactions for Large-Scale Synchronous Peer Learning
"... This research investigates how to introduce synchronous in-teractive peer learning into an online setting appropriate both for crowdworkers (learning new tasks) and students in mas-sive online courses (learning course material). We present an interaction framework in which groups of learners are for ..."
Abstract
- Add to MetaCart
(Show Context)
This research investigates how to introduce synchronous in-teractive peer learning into an online setting appropriate both for crowdworkers (learning new tasks) and students in mas-sive online courses (learning course material). We present an interaction framework in which groups of learners are formed on demand and then proceed through a sequence of activi-ties that include synchronous group discussion about learner-generated responses. Via controlled experiments with crowd-workers, we show that discussing challenging problems leads to better outcomes than working individually, and incentiviz-ing people to help one another yields still better results. We then show that providing a mini-lesson in which workers con-sider the principles underlying the tested concept and justify their answers leads to further improvements. Combining the mini-lesson with the discussion of the multiple-choice ques-tion leads to significant improvements on that question. We also find positive subjective responses to the peer interactions, suggesting that discussions can improve morale in remote work or learning settings.
Structuring Interactions for Large-Scale Synchronous Peer Learning
"... This research investigates how to introduce synchronous in-teractive peer learning into an online setting appropriate both for crowdworkers (learning new tasks) and students in mas-sive online courses (learning course material). We present an interaction framework in which groups of learners are for ..."
Abstract
- Add to MetaCart
(Show Context)
This research investigates how to introduce synchronous in-teractive peer learning into an online setting appropriate both for crowdworkers (learning new tasks) and students in mas-sive online courses (learning course material). We present an interaction framework in which groups of learners are formed on demand and then proceed through a sequence of activi-ties that include synchronous group discussion about learner-generated responses. Via controlled experiments with crowd-workers, we show that discussing challenging problems leads to better outcomes than working individually, and incentiviz-ing people to help one another yields still better results. We then show that providing a mini-lesson in which workers con-sider the principles underlying the tested concept and justify their answers leads to further improvements. Combining the mini-lesson with the discussion of the multiple-choice ques-tion leads to significant improvements on that question. We also find positive subjective responses to the peer interactions, suggesting that discussions can improve morale in remote work or learning settings.
