Abstract

We examine methods for clustering in high dimensions. In the first part of the paper, we perform an experimental comparison between three batch clustering algorithms: the Expectation–Maximization (EM) algorithm, a “winner take all ” version of the EM algorithm reminiscent of the K-means algorithm, and model-based hierarchical agglomerative clustering. We learn naive-Bayes models with a hidden root node, using high-dimensional discrete-variable data sets (both real and synthetic). We find that the EM algorithm significantly outperforms the other methods, and proceed to investigate the effect of various initialization schemes on the final solution produced by the EM algorithm. The initializations that we consider are (1) parameters sampled from an uninformative prior, (2) random perturbations of the marginal distribution of the data, and (3) the output of hierarchical agglomerative clustering. Although the methods are substantially different, they lead to learned models that are strikingly similar in quality. 1

Citations

6232	Maximum likelihood from incomplete data via the EM algorithm - Dempster, Laird, et al. - 1977
787	Human Behavior and the Principle of Least-Effort - Zipf - 1949
416	Bayesian classification (AutoClass): Theory and results - Cheeseman, Stutz - 1996
253	Raftery A: Model-based Gaussian and non-Gaussian clustering. Biometrics - Banfield - 1993
155	Efficient approximations for the marginal likelihood of Bayesian networks with hidden variables - Chickering, Heckerman - 1997
82	A classification EM algorithm for clustering and two stochastic versions - Celleux, Govaert - 1992
58	An Introduction to Generalized Linear Models - Dobson - 1990
47	Update rules for parameter estimation in Bayesian networks - Bauer, Koller, et al. - 1997
43	Algorithms for model-based Gaussian hierarchical clustering - Fraley - 1998
32	Human Behavior and the Principle of Least E ort - Zipf - 1949
29	E cient approximations for the marginal likelihood of Bayesian networks with hidden variables - Chickering, Heckerman - 1997
24	Learning mixtures of DAG models - Thiesson, Meek, et al. - 1998
22	Does the wake-sleep algorithm produce good density estimators - Frey, Hinton, et al. - 1996
22	Baysian classi cation (autoclass): Theory and results - Cheeseman, Stutz - 1996
21	Accelerated quantification of Bayesian networks with incomplete data - Thiesson - 1995
20	Model-based Gaussian and non-Gaussian clustering, Biometrics 49: 803–821 - eld, D, et al. - 1993
8	Latent class models,” in Handbook of statistical modeling for the social and behavioral sciences - Clogg - 1995
7	Accelerated quanti cation of Bayesian networks with incomplete data - Thiesson - 1995
4	A classi cation EM algorithm for clustering and two stochastic versions - Celeux, Govaert - 1992