Abstract

Abstract. In many data mining projects the data to be analysed contains personal information, like names and addresses. Cleaning and preprocessing of such data likely involves deduplication or linkage with other data, which is often challenged by a lack of unique entity identifiers. In recent years there has been an increased research effort in data linkage and deduplication, mainly in the machine learning and database communities. Publicly available test data with known deduplication or linkage status is needed so that new linkage algorithms and techniques can be tested, evaluated and compared. However, publication of data containing personal information is normally impossible due to privacy and confidentiality issues. An alternative is to use artificially created data, which has the advantages that content and error rates can be controlled, and the deduplication or linkage status is known. Controlled experiments can be performed and replicated easily. In this paper we present a freely available data set generator capable of creating data sets containing names, addresses and other personal information. 1

Citations

311	A theory for record linkage - Fellegi, Sunter - 1969
294	Techniques for automatically correcting words in text - Kukich - 1992
254	Merge/Purge Problem for Large Databases - Hernandez, Stolfo - 1995
243	A comparison of string distance metrics for namematching tasks - Cohen, Ravikumar, et al. - 2003
200	Efficient clustering of high-dimensional data sets with application to reference matching - McCallum, Nigam, et al. - 2000
180	R.J.: Adaptive Duplicate Detection Using Learnable String Similarity Measures - Bilenko, Mooney - 2003
161	Interactive deduplication using active learning - Sarawagi, Bhamidipaty - 2002
156	A technique for computer detection and correction of spelling errors - Damerau - 1964
119	Approximate string matching - Hall, Dowling - 1980
96	Learning to match and cluster large high-dimensional data sets for data integration - Cohen, Richman - 2002
95	UCI repository of machine learning datasets - Merz, Murphy - 1998
92	Learning domain-independent string transformation weights for high accuracy object identification - Tejada, Knoblock, et al. - 2002
51	Automatic spelling correction in scientific and scholarly text - Pollock, Zamora - 1984
43	R.: Robust identification of fuzzy duplicates - Chaudhuri, Ganti, et al. - 2005
29	A hierarchical graphical model for record linkage - Ravikumar, Cohen - 2004
24	R.J.: On evaluation and training-set construction for duplicate detection - Bilenko, Mooney - 2003
23	M.: Febrl - A Parallel Open Source Data Linkage System - Christen, Churches, et al. - 2004
19	Two approaches to handling noisy variation in text mining. Proceedings of the ICML-2002 workshop on text learning (TextML'2002 - Nahm, Bilenko, et al. - 1996
17	Automated record matching in cooperative information systems - Bertolazzi, Santis, et al. - 2003
15	Adaptive filtering for efficient record linkage - Gu, Baxter - 2004
13	String edit analysis for merging databases - Zhu, Unger - 2000
9	An adaptive string comparator for record linkage - Yancey - 2004
4	Elmagarmid, A.K.: TAILOR: A Record Linkage Toolbox - Elfeky, Verykios - 2002