Results 1  10
of
20
1 Parallel Spectral Clustering in Distributed Systems
"... Spectral clustering algorithms have been shown to be more effective in finding clusters than some traditional algorithms such as kmeans. However, spectral clustering suffers from a scalability problem in both memory use and computational time when the size of a data set is large. To perform cluster ..."
Abstract

Cited by 63 (1 self)
 Add to MetaCart
(Show Context)
Spectral clustering algorithms have been shown to be more effective in finding clusters than some traditional algorithms such as kmeans. However, spectral clustering suffers from a scalability problem in both memory use and computational time when the size of a data set is large. To perform clustering on large data sets, we investigate two representative ways of approximating the dense similarity matrix. We compare one approach by sparsifying the matrix with another by the Nyström method. We then pick the strategy of sparsifying the matrix via retaining nearest neighbors and investigate its parallelization. We parallelize both memory use and computation on distributed computers. Through
Sampling Methods for the Nyström Method
 JOURNAL OF MACHINE LEARNING RESEARCH
"... The Nyström method is an efficient technique to generate lowrank matrix approximations and is used in several largescale learning applications. A key aspect of this method is the procedure according to which columns are sampled from the original matrix. In this work, we explore the efficacy of a v ..."
Abstract

Cited by 26 (2 self)
 Add to MetaCart
The Nyström method is an efficient technique to generate lowrank matrix approximations and is used in several largescale learning applications. A key aspect of this method is the procedure according to which columns are sampled from the original matrix. In this work, we explore the efficacy of a variety of fixed and adaptive sampling schemes. We also propose a family of ensemblebased sampling algorithms for the Nyström method. We report results of extensive experiments that provide a detailed comparison of various fixed and adaptive sampling techniques, and demonstrate the performance improvement associated with the ensemble Nyström method when used in conjunction with either fixed or adaptive sampling schemes. Corroborating these empirical findings, we present a theoretical analysis of the Nyström method, providing novel error bounds guaranteeing a better convergence rate of the ensemble Nyström method in comparison to the standard Nyström method.
Clustered Nyström method for large scale manifold learning and dimension reduction
 IEEE Transactions on Neural Networks
, 2010
"... Abstract — Kernel (or similarity) matrix plays a key role in many machine learning algorithms such as kernel methods, manifold learning, and dimension reduction. However, the cost of storing and manipulating the complete kernel matrix makes it infeasible for large problems. The Nyström method is a p ..."
Abstract

Cited by 24 (6 self)
 Add to MetaCart
(Show Context)
Abstract — Kernel (or similarity) matrix plays a key role in many machine learning algorithms such as kernel methods, manifold learning, and dimension reduction. However, the cost of storing and manipulating the complete kernel matrix makes it infeasible for large problems. The Nyström method is a popular samplingbased lowrank approximation scheme for reducing the computational burdens in handling large kernel matrices. In this paper, we analyze how the approximating quality of the Nyström method depends on the choice of landmark points, and in particular the encoding powers of the landmark points in summarizing the data. Our (nonprobabilistic) error analysis justifies a “clustered Nyström method ” that uses the kmeans clustering centers as landmark points. Our algorithm can be applied to scale up a wide variety of algorithms that depend on the eigenvalue decomposition of kernel matrix (or its variant), such as kernel principal component analysis, Laplacian eigenmap, spectral clustering, as well as those involving kernel matrix inverse such as leastsquares support vector machine and Gaussian process regression. Extensive experiments demonstrate the competitive performance of our algorithm in both accuracy and efficiency. Index Terms — Dimension reduction, eigenvalue decomposition, kernel matrix, lowrank approximation, manifold learning,
DensityWeighted Nyström Method for Computing Large Kernel EigenSystems
, 2009
"... The Nyström method is a wellknown samplingbased technique for approximating the eigensystem of large kernel matrices. However, the chosen samples in the Nyström method are all assumed to be of equal importance, which deviates from the integral equation that defines the kernel eigenfunctions. Moti ..."
Abstract

Cited by 20 (4 self)
 Add to MetaCart
(Show Context)
The Nyström method is a wellknown samplingbased technique for approximating the eigensystem of large kernel matrices. However, the chosen samples in the Nyström method are all assumed to be of equal importance, which deviates from the integral equation that defines the kernel eigenfunctions. Motivated from this observation, we extend the Nyström method to a more general, densityweighted version. We show that by introducing the probability density function as a natural weighting scheme, the approximation of the eigensystem can be greatly improved. An efficient algorithm is proposed to enforce such weighting in practice, which has the same complexity as the original Nyström method and hence notably cheaper than several other alternatives. Experiments on kernel principal component analysis, spectral clustering and image segmentation all demonstrate the encouraging performance of our algorithm.
Robust PathBased Spectral Clustering with Application to Image Segmentation
"... Spectral clustering and pathbased clustering are two recently developed clustering approaches that have delivered impressive results in a number of challenging clustering tasks. However, they are not robust enough against noise and outliers in the data. In this paper, based on Mestimation from rob ..."
Abstract

Cited by 15 (2 self)
 Add to MetaCart
(Show Context)
Spectral clustering and pathbased clustering are two recently developed clustering approaches that have delivered impressive results in a number of challenging clustering tasks. However, they are not robust enough against noise and outliers in the data. In this paper, based on Mestimation from robust statistics, we develop a robust pathbased spectral clustering method by defining a robust pathbased similarity measure for spectral clustering. Our method is significantly more robust than spectral clustering and pathbased clustering. We have performed experiments based on both synthetic and realworld data, comparing our method with some other methods. In particular, color images from the Berkeley Segmentation Dataset and Benchmark are used in the image segmentation experiments. Experimental results show that our method consistently outperforms other methods due to its higher robustness.
On landmark selection and sampling in highdimensional data analysis. arXiv:0906.4582v1[stat.ML
, 2009
"... In recent years, the spectral analysis of appropriately defined kernel matrices has emerged as a principled way to extract the lowdimensional structure often prevalent in highdimensional data. Here we provide an introduction to spectral methods for linear and nonlinear dimension reduction, emphasi ..."
Abstract

Cited by 11 (3 self)
 Add to MetaCart
In recent years, the spectral analysis of appropriately defined kernel matrices has emerged as a principled way to extract the lowdimensional structure often prevalent in highdimensional data. Here we provide an introduction to spectral methods for linear and nonlinear dimension reduction, emphasizing ways to overcome the computational limitations currently faced by practitioners with massive datasets. In particular, a data subsampling or landmark selection process is often employed to construct a kernel based on partial information, followed by an approximate spectral analysis termed the Nyström extension. We provide a quantitative framework to analyse this procedure, and use it to demonstrate algorithmic performance bounds on a range of practical approaches designed to optimize the landmark selection process. We compare the practical implications of these bounds by way of realworld examples drawn from the field of computer vision, whereby lowdimensional manifold structure is shown to emerge from highdimensional video data streams.
Blockquantized kernel matrix for fast spectral embedding
 In Proceedings of International Conference on Machine learning (ICML’06
, 2006
"... Eigendecomposition of kernel matrix is an indispensable procedure in many learning and vision tasks. However, the cubic complexity O(N 3) is impractical for large problem, where N is the data size. In this paper, we propose an efficient approach to solve the eigendecomposition of the kernel matrix W ..."
Abstract

Cited by 7 (4 self)
 Add to MetaCart
(Show Context)
Eigendecomposition of kernel matrix is an indispensable procedure in many learning and vision tasks. However, the cubic complexity O(N 3) is impractical for large problem, where N is the data size. In this paper, we propose an efficient approach to solve the eigendecomposition of the kernel matrix W. The idea is to approximate W with W that is composed of m 2 constant blocks. The eigenvectors of W, which can be solved in O(m 3) time, is then used to recover the eigenvectors of the original kernel matrix. The complexity of our method is only O(mN + m 3), which scales more favorably than stateoftheart low rank approximation and sampling based approaches (O(m 2 N + m 3)), and the approximation quality can be controlled conveniently. Our method demonstrates encouraging scaling behaviors in experiments of image segmentation (by spectral clustering) and kernel principal component analysis. 1.
Nonparametric SemiSupervised Learning for Network Intrusion Detection: Combining Performance Improvements with Realistic InSitu Training
"... A barrier to the widespread adoption of learningbased network intrusion detection tools is the insitu training requirements for effective discrimination of malicious traffic. Supervised learning techniques necessitate a quantity of labeled examples that is often intractable, and at best costp ..."
Abstract

Cited by 5 (1 self)
 Add to MetaCart
(Show Context)
A barrier to the widespread adoption of learningbased network intrusion detection tools is the insitu training requirements for effective discrimination of malicious traffic. Supervised learning techniques necessitate a quantity of labeled examples that is often intractable, and at best costprohibitive. Recent advances in semisupervised techniques have demonstrated the ability to generalize knowledge based on a significantly smaller number of provided examples. In network intrusion detection, placing reasonable requirements on the number of training examples provides realistic expectations that a learningbased system can be trained in the environment where it will be deployed. This insitu training is necessary to ensure that the assumptions associated with the learning process hold, and thereby support a reasonable belief in the generalization ability of the resulting model. In this paper, we describe the application of a carefully selected nonparametric, semisupervised learning algorithm to the network intrusion problem, and compare the performance to other model types using featurebased data derived from an operational network. We demonstrate dramatic performance improvements over supervised learning and anomaly detection in discriminating real, previously unseen, malicious network traffic while generating an order of magnitude fewer false alerts than any alternative, including a signature IDS tool deployed on the same network.
Approximation of Positive Semidefinite Matrices Using the Nyström Method
, 2011
"... Positive semidefinite matrices arise in a variety of fields, including statistics, signal processing, and machine learning. Unfortunately, when these matrices are highdimensional and/or must be operated upon many times, expensive calculations such as the spectral decomposition quickly become a comp ..."
Abstract

Cited by 1 (0 self)
 Add to MetaCart
(Show Context)
Positive semidefinite matrices arise in a variety of fields, including statistics, signal processing, and machine learning. Unfortunately, when these matrices are highdimensional and/or must be operated upon many times, expensive calculations such as the spectral decomposition quickly become a computational bottleneck. A common alternative is to replace the original positive semidefinite matrices with lowrank approximations whose spectral decompositions can be more easily computed. In this thesis, we develop approaches based on the Nyström method, which approximates a positive semidefinite matrix using a datadependent orthogonal projection. As the Nyström approximation is conditioned on a given principal submatrix of its argument, it essentially recasts lowrank approximation as a subset selection problem. We begin by deriving the Nyström approximation and developing a number of fundamental results, including new characterizations of its spectral properties and approximation error. We then address the problem of subset selection through a study of randomized sampling algorithms. We provide new bounds for the approximation error under uniformly random