Abstract

The detection of duplicate tuples, corresponding to the same real-world entity, is an important task in data integration and cleaning. While many techniques exist to identify such tuples, the merging or elimination of duplicates can be a difficult task that relies on ad-hoc and often manual solutions. We propose a complementary approach that permits declarative query answering over duplicated data, where each duplicate is associated with a probability of being in the clean database. We rewrite queries over a database containing duplicates to return each answer with the probability that the answer is in the clean database. Our rewritten queries are sensitive to the semantics of duplication and help a user understand which query answers are most likely to be present in the clean database. The semantics that we adopt is independent of the way the probabilities are produced, but is able to effectively exploit them during query answering. In the absence of external knowledge that associates each database tuple with a probability, we offer a technique, based on tuple summaries, that automates this task. We experimentally study the performance of our rewritten queries. Our studies show that the rewriting does not introduce a significant overhead in query execution time. This work is done in the context of the ConQuer project at the University of Toronto, which focuses on the efficient management of inconsistent and dirty databases. 1

Citations

6514	Elements of Information Theory - Cover, Thomas - 1991
311	A theory for record linkage - Fellegi, Sunter - 1969
275	Efficient query evaluation on probabilistic databases - Dalvi, Suciu - 2007
227	Consistent query answers in inconsistent databases - Arenas, Bertossi, et al. - 1999
200	Efficient clustering of high-dimensional data sets with application to reference matching - McCallum, Nigam, et al. - 2000
195	The management of probabilistic data - Barbará, Garcia-Molina, et al. - 1992
180	R.J.: Adaptive Duplicate Detection Using Learnable String Similarity Measures - Bilenko, Mooney - 2003
172	WE: The state of record linkage and current research problems - Winkler - 1999
161	Interactive deduplication using active learning - Sarawagi, Bhamidipaty - 2002
151	Real-World Data is Dirty: Data Cleansing and the Merge/Purge Problem - Hernandez, Stolfo - 1998
149	A probabilistic relational algebra for the integration of information retrieval and database systems - Fuhr, Rölleke - 1997
145	Probview: a flexible probabilistic database system - Lakshmanan, Leone, et al. - 1997
119	The theory of probabilistic databases - Cavallo, Pittarelli - 1987
97	Eliminating fuzzy duplicates in data warehouses - Ananthakrishna, Chaudhuri, et al. - 2002
96	On the decidability and complexity of query answering over inconsistent and incomplete databases - Cali, Lembo, et al. - 2003
86	Declarative data cleaning: Language, model, and algorithms - Galhardas, Florescu, et al. - 2001
67	Minimal-change integrity maintenance using tuple deletions - Chomicki, Marcinkowski - 2005
27	Clean answers over dirty databases - Andritsos, Fuxman, et al. - 2006
19	Information-theoretic tools for mining database structure from large data sets - Andritsos, Miller, et al. - 2004
19	Deduplication and Group Detection Using Links - Battacharya, Getoor - 2004
11	LIMBO: Scalable clustering of categorical data - Andritsos, Tsaparas, et al. - 2004