Abstract

We propose a new algorithm capable of partitioning a set of documents or other samples based on an embedding in a high dimensional Euclidean space (i.e. in which every document is a vector of real numbers). The method is unusual in that it is divisive, as opposed to agglomerative, and operates by repeatedly splitting clusters into smaller clusters. The splits are not based on any distance or similarity measure. The documents are assembled in to a matrix which is very sparse. It is this sparsity that permits the algorithm to be very efficient. The performance of the method is illustrated with a set of text documents obtained from the World Wide Web. Some possible extensions are proposed for further investigation.

Citations

3340	Pattern Classification and Scene Analysis - Duda, Hart - 1973
2699	MJ: Introduction to Modern Information Retrieval - Salton, McGill - 1986
1753	Dubes, Algorithms for Clustering Data - Jain, C - 1988
525	Cluster Analysis for Applications - Anderberg - 1973
519	Scatter/gather: A cluster-based approach to browsing large document collections - Cutting, Pedersen, et al. - 1992
511	DM: Statistical Analysis of Finite Mixture Distributions Chichester - Titterington - 1985
450	Using Linear Algebra for Intelligent Information Retrieval - Berry, Dumais, et al. - 1995
427	Information retrieval data structures and algorithms - Frakes, Baeza-Yates - 1992
416	Bayesian classification (AutoClass): Theory and results - Cheeseman, Stutz - 1996
57	WebACE: A web agent for document categorization and exploration - Han, Boley, et al. - 1998
41	A sentence-to-sentence clustering procedure for pattern analysis - Lu, Fu - 1987
33	Pattern Recognition Engineering - Nadler, Smith - 1993
20	Web page categorization and feature selection using association rule and principal component clustering - Moore, Han, et al. - 1997