Abstract

Abstract. Data cleaning and integration is typically the most expensive step in the KDD process. A key part, known as record linkage or de-duplication, is identifying which records in a database refer to the same entities. This problem is traditionally solved separately for each candidate record pair (followed by transitive closure). We propose to use instead a multi-relational approach, performing simultaneous inference for all candidate pairs, and allowing information to propagate from one candidate match to another via the attributes they have in common. Our formulation is based on conditional random fields, and allows an optimal solution to be found in polynomial time using a graph cut algorithm. Parameters are learned using a voted perceptron algorithm. Experiments on real and synthetic databases show that multi-relational record linkage outperforms the standard approach. 1

Citations

2699	Introduction to Modern Information Retrieval - Salton, McGill - 1983
1548	F: Conditional Random Fields: Probabilistic models for segmenting and labeling sequence data - Lafferty, McCallum, et al. - 2001
820	A: Categorical data analysis - Agresti - 1990
340	BDiscriminative training methods for hidden Markov models - Collins
311	A Theory of Record Linkage - FELLEGI, SUNTER - 1969
276	D: Discriminative probabilistic models for relational data - Taskar, Abbeel, et al. - 2002
254	Merge/Purge Problem for Large Databases - Hernandez, Stolfo - 1995
233	Exact maximum a posteriori estimation for binary images - Greig, Porteous, et al. - 1989
200	Efficient clustering of high-dimensional data sets with application to reference matching - McCallum, Nigam, et al. - 2000
180	R.J.: Adaptive Duplicate Detection Using Learnable String Similarity Measures - Bilenko, Mooney - 2003
172	WE: The state of record linkage and current research problems - Winkler - 1999
161	Interactive deduplication using active learning - Sarawagi, Bhamidipaty - 2002
154	An efficient domain-independent algorithm for detecting approximately duplicate database records - Monge, Elkan - 1997
130	Identity uncertainty and citation matching - Pasula, Marthi, et al. - 2002
106	Automatic linkage of vital records - Newcombe, Kennedy, et al. - 1959
96	Learning to match and cluster large high-dimensional data sets for data integration - Cohen, Richman - 2002
92	Learning domain-independent string transformation weights for high accuracy object identification - Tejada, Knoblock, et al. - 2002
50	Hardening soft information sources - Cohen, Kautz, et al. - 2000
14	Object consolidation by graph partitioning with a conditionally trained distance metric - MCCALLUM, WELLNER