## Finding the number of clusters in a data set: An information theoretic approach (2003)

Venue: | Journal of the American Statistical Association |

Citations: | 49 - 1 self |

### BibTeX

@ARTICLE{Sugar03findingthe,

author = {Catherine A. Sugar and Gareth and M. James},

title = {Finding the number of clusters in a data set: An information theoretic approach},

journal = {Journal of the American Statistical Association},

year = {2003},

volume = {98},

pages = {750--763}

}

### Years of Citing Articles

### OpenURL

### Abstract

One of the most difficult problems in cluster analysis is the identification of the number of groups in a data set. Most previously suggested approaches to this problem are either somewhat ad hoc or require parametric assumptions and complicated calculations. In this paper we develop a simple yet powerful non-parametric method for choosing the number of clusters based on distortion, a quantity that measures the average distance, per dimension, between each observation and its closest cluster center. Our technique is computationally efficient and straightforward to implement. We demonstrate empirically its effectiveness, not only for choosing the number of clusters but also for identifying underlying structure, on a wide range of simulated and real world data sets. In addition, we give a rigorous theoretical justification for the method based on information theoretic ideas. Specifically, results from the subfield of electrical engineering known as rate distortion theory allow us to describe the behavior of the distortion in both the presence and absence of clustering. Finally, we note that these ideas potentially can be extended to a wide range of other statistical model selection problems. 1