Although existing work has explored both information extraction and community content creation, most research has focused on them in isolation. In contrast, we see the greatest leverage in the synergistic pairing of these methods as two interlocking feedback cycles. This paper explores the potential synergy promised if these cycles can be made to accelerate each other by exploiting the same edits to advance both community content creation and learning-based information extraction. We examine our proposed synergy in the context of Wikipedia infoboxes and the Kylin information extraction system. After developing and refining a set of interfaces to present the verification of Kylin extractions as a non-primary task in the context of Wikipedia articles, we develop an innovative use of Web search advertising services to study people engaged in some other primary task. We demonstrate our proposed synergy by analyzing our deployment from two complementary perspectives: (1) we show we accelerate community content creation by using Kylin’s information extraction to significantly increase the likelihood that a person visiting a Wikipedia article as a part of some other primary task will spontaneously choose to help improve the article’s infobox, and (2) we show we accelerate information extraction by using contributions collected from people interacting with our designs to significantly improve Kylin’s extraction performance. ACM Classification:

Although machine learning is becoming commonly used in today’s software, there has been little research into how end users might interact with machine learning systems, beyond communicating simple "right/wrong " judgments. If the users themselves could work hand-in-hand with machine learning systems, the users ’ understanding and trust of the system could improve and the accuracy of learning systems could be improved as well. We conducted three experiments to understand the potential for rich interactions between users and machine learning systems. The first experiment was a think-aloud study that investigated users ’ willingness to interact with machine learning reasoning, and what kinds of feedback users might give to machine learning systems. We then investigated the viability of introducing such feedback into machine learning systems, specifically, how to incorporate some of these types of user feedback into machine learning systems, and what their impact was on the accuracy of the system. Taken together, the results of our experiments show that supporting rich interactions between users and machine learning systems is feasible for both user and machine. This shows the potential of rich humancomputer collaboration via on-the-spot interactions as a promising direction for machine learning systems and users to collaboratively share intelligence.

The potential for machine learning systems to improve via a mutually beneficial exchange of information with users has yet to be explored in much detail. Previously, we found that users were willing to provide a generous amount of rich feedback to machine learning systems, and that the types of some of this rich feedback seem promising for assimilation by machine learning algorithms. Following up on those findings, we ran an experiment to assess the viability of incorporating real-time keyword-based feedback in initial training phases when data is limited. We found that rich feedback improved accuracy but an initial unstable period often caused large fluctuations in classifier behavior. Participants were able to give feedback by relying heavily on system communication in order to respond to changes. The results show that in order to benefit from the user’s knowledge, machine learning systems must be able to absorb keyword-based rich feedback in a graceful manner and provide clear explanations of their predictions. Author Keywords Machine learning, user feedback. ACM CLASSIFICATION KEYWORDS H.5.2 [Information interfaces and presentation (e.g., HCI)] User Interfaces: Theory and methods. H.1.2 [Models and Principles]: User/Machine Systems: Human information processing, Human factors.

Existing template-independent web data extraction approaches adopt highly ineffective decoupled strategies—attempting to do data record detection and attribute labeling in two separate phases. In this paper, we propose an integrated web data extraction paradigm with hierarchical models. The proposed model is called Dynamic Hierarchical Markov Random Fields (DHMRFs). DHMRFs take structural uncertainty into consideration and define a joint distribution of both model structure and class labels. The joint distribution is an exponential family distribution. As a conditional model, DHMRFs relax the independence assumption as made in directed models. Since exact inference is intractable, a variational method is developed to learn the model’s parameters and to find the MAP model structure and label assignments. We apply DHMRFs to a real-world web data extraction task. Experimental results show that: (1) integrated web data extraction models can achieve significant improvements on both record detection and attribute labeling compared to decoupled models; (2) in diverse web data extraction DHMRFs can potentially address the blocky artifact issue which is suffered by fixed-structured hierarchical models.

n The RADAR project developed a large multiagent system with a mixed-initiative user interface designed to help office workers cope with e-mail overload. Most RADAR agents observe experts performing tasks and then assist other users who are performing similar tasks. The interaction design for RADAR focused on developing user interfaces that allowed the intelligent functionality to improve the user’s workflow without frustrating the user when the system’s suggestions were either unhelpful or simply incorrect. For example, with regard to autonomy, the RADAR agents were allowed much flexibility in selecting ways to assist the user but were restricted from taking actions that would

We present two studies that evaluate the accuracy of human responses to an intelligent agent’s data classification questions. Prior work has shown that agents can elicit accurate human responses, but the applications vary widely in the data features and prediction information they provide to the labelers when asking for help. In an initial analysis of this work, we found the five most popular features, namely uncertainty, amount and level of context, prediction of an answer, and request for user feedback. We propose that there is a set of these data features and prediction information that maximizes the accuracy of labeler responses. In our first study, we compare accuracy of users of an activity recognizer labeling their own data across the dimensions. In the second study, participants were asked to classify a stranger’s emails into folders and strangers ’ work activities by interruptibility. We compared the accuracy of the responses to the users ’ self-reports across the same five dimensions. We found very similar combinations of information (for users and strangers) that led to very accurate responses as well as more feedback that the agents could use to refine their predictions. We use these results for insight into the information that help labelers the most.

Abstract not found

the degree of Doctor of Philosophy. Human computation is the study of systems where humans perform a major part of the computation or are an integral part of the overall computational process. The ESP Game, for example, is a human computation system that maps images to tags, by engaging humans to play a game in which they are rewarded each time they agree on a description for an image. It was shown that these so-called Games with a Purpose are a reliable way to quickly collect millions of accurate image descriptors, which can then used to index images and facilitate search. However, most existing human computation systems operate without any machine intervention. Likewise, very few supervised learning systems are taking advantage of these powerful new platforms to elicit help from human teachers. It is therefore largely unknown what more a human computation system can achieve with machines in the loop. This thesis is centered around the problem of attribute learning – using the joint effort of human game players and machine learning algorithms to determine that a piece of music is “soothing”, that the bird in an image “has a red beak”, or that Ernest Hemingway is an “Nobel Prize winning author”. In particular, our work focuses on two aspects of the problem – how to acquire attributes and attribute values from human computers using incentive-compatible game mechanisms, and what active learning strategies to employ for attribute and attribute value acquisition.

Abstract not found

shailesh.saroha