DMCA
Real-world Data is Dirty: Data Cleansing and The Merge/Purge Problem (1998)
Cached
Download Links
- [www.lans.ece.utexas.edu]
- [www.cs.columbia.edu]
- DBLP
Other Repositories/Bibliography
| Venue: | DATA MINING AND KNOWLEDGE DISCOVERY |
| Citations: | 245 - 0 self |
BibTeX
@ARTICLE{Hernández98real-worlddata,
author = {Mauricio A. Hernández and Salvatore J. Stolfo},
title = {Real-world Data is Dirty: Data Cleansing and The Merge/Purge Problem},
journal = {DATA MINING AND KNOWLEDGE DISCOVERY},
year = {1998},
volume = {2},
pages = {9--37}
}
Share
 |
 |
 |
 |
||
OpenURL
Abstract
The problem of merging multiple databases of information about common entities is frequently encountered in KDD and decision support applications in large commercial and government organizations. The problem we study is often called the Merge/Purge problem and is difficult to solve both in scale and accuracy. Large repositories of data typically have numerous duplicate information entries about the same entities that are difficult to cull together without an intelligent "equational theory" that identifies equivalent items by a complex, domain-dependent matching process. We have developed a system for accomplishing this Data Cleansing task and demonstrate its use for cleansing lists of names of potential customers in a direct marketing-type application. Our results for statistically generated data are shown to be accurate and effective when processing the data multiple times using different keys for sorting on each successive pass. Combing results of individual passes using transitive c...
Keyphrases
merge purge problem real-world data data cleansing data multiple time government organization numerous duplicate information entry equivalent item potential customer multiple database large repository data cleansing task different key successive pas domain-dependent matching process decision support application intelligent equational theory direct marketing-type application common entity individual pass
