Mechanical Turk provides an on-demand source of human computation. This provides a tremendous opportunity to explore algorithms which incorporate human computation as a function call. However, various systems challenges make this difficult in practice, and most uses of Mechanical Turk post large numbers of independent tasks. TurKit is a toolkit for prototyping and exploring truly algorithmic human computation, while maintaining a straight-forward imperative programming style. We present the crash-andrerun programming model that makes TurKit possible, along with a variety of applications for human computation algorithms. We also present a couple case studies of TurKit used for real experiments outside our lab. ACM Classification: H5.2 [Information interfaces and

A musically meaningful vocabulary is one of the keystones in building a computer audition system that can model the semantics of audio content. If a word in the vocabulary is inconsistently used by human annotators, or the word is not clearly represented by the underlying acoustic representation, the word can be considered as noisy and should be removed from the vocabulary to denoise the modeling process. This paper proposes an approach to construct a vocabulary of predictive semantic concepts based on sparse canonical component analysis (sparse CCA). Experimental results illustrate that, by identifying musically meaningful words, we can improve the performance of a previously proposed computer audition system for music annotation and retrieval. 1

Abstract not found

This paper presents an approach to acquire knowledge from Wikipedia categories and the category network. Many Wikipedia categories have complex names which reflect human classification and organizing instances, and thus encode knowledge about class attributes, taxonomic and other semantic relations. We decode the names and refer back to the network to induce relations between concepts in Wikipedia represented through pages or categories. The category structure allows us to propagate a relation detected between constituents of a category name to numerous concept links. The results of the process are evaluated against ResearchCyc and a subset also by human judges. The results support the idea that Wikipedia category names are a rich source of useful and accurate knowledge.

The collaboration landscape evolves rapidly by allowing people to participate in ad-hoc and process-centric collaborations. Thus, it is important to support humans in managing highly dynamic and complex interactions. The problem currently with managing interactions is that humans are unable to specify different interaction interfaces for various collaborations, nor able to indicate their availability to participate in collaborations. This paper introduces the Human-provided Services (HpS) framework, which allows users to provide services based on their skills and expertise. Such services can be used by human actors and software services in both ad-hoc and process-centric collaborations. With the HpS framework, people can offer multiple services and manage complex interactions, while requesters can find the right experts and available users for performing specific tasks. In this paper, we present the HpS middleware, which is the core of the HpS framework. We show how HpS services can be used in Web-scale ad-hoc collaboration scenarios. 1

Abstract: Annotated corpora of the size needed for modern computational linguistics research cannot be created by small groups of hand annotators. One solution is to exploit collaborative work on the Web and one way to do this is through games like the ESP game. Applying this methodology however requires developing methods for teaching subjects the rules of the game and evaluating their contribution while maintaining the game entertainment. In addition, applying this method to linguistic annotation tasks like anaphoric annotation requires developing methods for presenting text and identifying the components of the text that need to be annotated. In this paper we

Music listeners frequently use words to describe music. Personalized music recommendation systems such as Last.fm and Pandora rely on manual annotations (tags) as a mechanism for querying and navigating large music collections. A well-known issue in such recommendation systems is known as the cold-start problem: it is not possible to recommend new songs/tracks until those songs/tracks have been manually annotated. Automatic tag annotation based on content analysis is a potential solution to this problem and has recently been gaining attention. We describe how stacked generalization can be used to improve the performance of a state-of-the-art automatic tag annotation system for music based on audio content analysis and report results on two publicly available datasets.

In this paper, we provide a simple game-theoretic model of an online question and answer forum. We focus on factual questions in which user responses aggregate while a question remains open. Each user has a unique piece of information and can decide when to report this information. The asker prefers to receive information sooner rather than later, and will stop the process when satisfied with the cumulative value of the posted information. We consider two distinct cases: a complements case, in which each successive piece of information is worth more to the asker than the previous one; and a substitutes case, in which each successive piece of information is worth less than the previous one. A best-answer scoring rule is adopted to model Yahoo! Answers, and is effective for substitutes information, where it isolates an equilibrium in which all users respond in the first round. But we find that this rule is ineffective for complements information, isolating instead an equilibrium in which all users respond in the final round. In addressing this, we demonstrate that an approval-voting scoring rule and a proportional-share scoring rule can enable the most efficient equilibrium with complements information, under certain conditions, by providing incentives for early responders as well as the user who submits the final answer.

In this paper, we describe a pilot summer workshop (CS4HS) held at Carnegie Mellon University in July 2006 for high school CS teachers to provide compelling material that the teachers can use in their classes to emphasize computational thinking and the many possibilities of computer science. Diversity and broadening participation was explicitly addressed throughout the workshop. We focused on broadening the image of what CS is – and who computer scientists are – since the reasons for underrepresentation in the field are very much the same as the reasons for the huge decline in interest. We describe the design of the workshop along with results from initial surveys and evaluations. Short-term evaluations show that this workshop was successful in changing the perception of CS for these teachers and giving them the impetus to include broader topics in their programming courses for the upcoming school year. Future surveys will track the long-term effect of this workshop.

We show how machine learning and inference can be harnessed to leverage the complementary strengths of humans and computational agents to solve crowdsourcing tasks. We construct a set of Bayesian predictive models from data and describe how the models operate within an overall crowdsourcing architecture that combines the efforts of people and machine vision on the task of classifying celestial bodies defined within a citizens ’ science project named Galaxy Zoo. We show how learned probabilistic models can be used to fuse human and machine contributions and to predict the behaviors of workers. We employ multiple inferences in concert to guide decisions on hiring and routing workers to tasks so as to maximize the efficiency of large-scale crowdsourcing processes based on expected utility.