Citation Context

...m is uncertain about predicting the label (e.g., reCAPTCHA project [1]). Some works have also used crowdsourcing for evaluation, but mostly for search relevance evaluation [2, 14] and entity matching =-=[9]-=-. In contrast, we use crowdsourcing to help evaluate the results of classification. Machine learning techniques have been broadly used for document and text classification [15, 24]. As the most descri...

Citation Context

...ata transformations [19, 48]. 3.4 Data Deduplication A number of systems have been developed for data deduplication, e.g., BigMatch [95], Tailor [32], Swoosh [10] AJAX [50], CrowdER [88] and Corleone =-=[57]-=-, as standalone tools, embedded packages in ETL systems, or crowdsourced systems. Criteria for developing such systems include (a) accuracy, to reduce false matches (false positives) and false non-mat...

Citation Context

...s simple text transformations, do not support crowd-based processing at scale, and cannot incorporate user feedback to optimize the physical implementation of the data cleaning sequences. Crowd-based =-=[7,13]-=- systems have been proposed to relieve the data analyst of the burden of rule specification or manual cleaning, but are usually specific to a single cleaning task (e.g., [5,7,11,12]), preventing end-t...

Citation Context

...about coreferent entities. Furthermore, Pujara et al. used probabilistic soft logic (PSL) to avoid scalability limitation of MLNs. Dong et al. [6] employed supervised machine learning methods for fusing distinct information sources by combining noisy extractions from the web with prior knowledge derived from existing knowledge repositories. However, all of above methods are automated algorithms and do not leverage the power of crowdsourcing. There also exist many research works that incorporate crowdsourcing into data and knowledge management, such as data cleaning [14, 16–19], record linkage [20, 21], schema matching [22–25], and knowledge acquisition [26, 27]. For example, Wang et al. [14] proposed CrowdER to solve the problem of entity resolution via crowdsourcing. Zhang et al. [16] used crowdsourcing to clean uncertain data. Chu et al. [18] proposed KATARA, a data cleaning system that utilizes the power of knowledge bases and crowdsourcing to clean tables. Demartini et al. [20] proposed ZenCrowd which uses a mixed human-machine workflow to solve the entity linking problem. Gokhale et al. [21] studied how to do hands-off crowdsourcing record linkage which requires no involvement of deve...

Citation Context

...for identifying matching record pairs, vary widely. In general, the workflow of crowdsourcing ER has been developed step by step. From the beginning, ER is proposed to be solved by crowdsourcing only, until now a much more complete workflow is formed by integrating different research (see Figure 4). In addition, these approaches focus on different problems. Many novel ideas and algorithms have been developed to optimize crowdsourcing ER. Gokhale et al. presented the Corleone approach and tried to leverage crowdsourcing for the whole process of ER, which is described as hands-off crowdsourcing [4]. In Subsection 3.1 the research leveraging crowdsourcing only for the identification of matching record pairs is described. The corleone approach, which leverages crowdsourcing for the whole process of ER, is then discussed in Subsection 3.2. 3.1 Crowdsourcing for Identifying Matching Record Pairs Figure 4 depicts a complete hybrid workflow for ER, which contains all proposals to optimize the workflow in crowdsourcing ER. Instead of letting crowds answer HITs that contain matching questions for all records, the first step in the complete workflow is to choose a proper machine-based method to ...

Citation Context

... interrupts the user during her primary data exploration task. Our goal is to clean a user’s visualization with minimal interruption. State-of-the-art techniques for deduplication use active learning =-=[93, 45]-=-, where one or 1http://www.cs.utexas.edu/users/ml/riddle 7Figure 1.3: Views, dirty (left) and clean (right), over Fodor ∪ Zagat restaurant datasets. more users label training examples, which enable th...

Citation Context

... (clustering) and order-by (ranking) [6, 7, 14, 15, 16, 17]. Crowdsourcing is also used in performing specific computational tasks such as entity resolution, data cleaning, and machine learning tasks =-=[4, 8, 11, 19, 20, 21]-=-. Including the crowd as an additional information and processing source in these ways has therefore required extensions to the query processing and optimization components of a traditional database a...

Citation Context

...fixing data error often requires manually inspecting data, which can quickly become costly and time-consuming. While crowdsourcing is an increasingly viable option for correcting some types of errors =-=[29, 22, 48, 24, 38, 4, 14]-=-, it comes at the significant cost of additional latency and the overhead of managing human workers. On the other hand, ignoring the effects of dirty data is potentially dangerous. Analysts have to ch...

Citation Context

...ges is given to humans, it would be relatively easy to identify that images (1) and (3) include cycling as the same activity. Recently while addressing this challenge, an array of new attempts (e.g., =-=[7, 15, 16, 17]-=-) have been made to incorporate a crowdsourcing framework into the ER process. The underlying premise of such attempts is that some ER tasks (e.g., Example 1) are inherently challenging for machines t...

Citation Context

...for machines, but rather trivial for humans. The number of crowdsourcing applications is tremendous, ranging from data acquisition [4], data integration [51], data mining [44], information extraction =-=[17]-=-, to information retrieval [50]. To facilitate the development of crowdsourcing applications, more than 70 crowdsourcing platforms such as Amazon Mechanical Turk (AMT) and CrowdFlower have been develo...