Interactive Deduplication using Active Learning

DMCA

Interactive Deduplication using Active Learning (2002)

Cached

Download Links

Other Repositories/Bibliography

DBLP

by Sunita Sarawagi , Anuradha Bhamidipaty

Citations:

242 - 5 self

BibTeX

@INPROCEEDINGS{Sarawagi02interactivededuplication,
    author = {Sunita Sarawagi and Anuradha Bhamidipaty},
    title = {Interactive Deduplication using Active Learning},
    booktitle = {},
    year = {2002},
    pages = {269--278},
    publisher = {}
}

OpenURL

Abstract

Deduplication is a key operation in integrating data from multiple sources. The main challenge in this task is designing a function that can resolve when a pair of records refer to the same entity in spite of various data inconsistencies. Most existing systems use hand-coded functions. One way to overcome the tedium of hand-coding is to train a classifier to distinguish between duplicates and non-duplicates. The success of this method critically hinges on being able to provide a covering and challenging set of training pairs that bring out the subtlety of the deduplication function. This is non-trivial because it requires manually searching for various data inconsistencies between any two records spread apart in large lists. We present our design of a learning-based deduplication system that uses a novel method of interactively discovering challenging training pairs using active learning. Our experiments on real-life datasets show that active learning signicantly reduces the number of instances needed to achieve high accuracy. We investigate various design issues that arise in building a system to provide interactive response, fast convergence, and interpretable output.

Keyphrases

active learning interactive deduplication various data inconsistency various design issue interactive response main challenge large list key operation interpretable output real-life datasets deduplication function hand-coded function fast convergence learning-based deduplication system high accuracy multiple source novel method

DMCA