DMCA
Efficient Clustering of High-Dimensional Data Sets with Application to Reference Matching (2000)
Cached
Download Links
| Citations: | 339 - 15 self |
BibTeX
@MISC{McCallum00efficientclustering,
author = {Andrew McCallum and Kamal Nigam and Lyle H. Ungar},
title = {Efficient Clustering of High-Dimensional Data Sets with Application to Reference Matching},
year = {2000}
}
Share
 |
 |
 |
 |
||
OpenURL
Abstract
Many important problems involve clustering large datasets. Although naive implementations of clustering are computationally expensive, there are established efficient techniques for clustering when the dataset has either (1) a limited number of clusters, (2) a low feature dimensionality, or (3) a small number of data points. However, there has been much less work on methods of efficiently clustering datasets that are large in all three ways at once|for example, having millions of data points that exist in many thousands of dimensions representing many thousands of clusters. We present a new technique for clustering these large, high-dimensional datasets. The key idea involves using a cheap, approximate distance measure to efficiently divide the data into overlapping subsets we call canopies. Then clustering is performed by measuring exact distances only between points that occur in a common canopy. Using canopies, large clustering problems that were formerly impossible become practical. Under r...
Keyphrases
efficient clustering reference matching high-dimensional data set many thousand data point naive implementation key idea many important problem large datasets low feature dimensionality efficient technique exact distance common canopy approximate distance measure small number limited number new technique high-dimensional datasets
