Results 1  10
of
40
Dynamics and generalization ability of LVQ algorithms
 Journal of Machine Learning Research
, 2006
"... Learning vector quantization (LVQ) schemes constitute intuitive, powerful classification heuristics with numerous successful applications but, so far, limited theoretical background. We study LVQ rigorously within a simplifying model situation: two competing prototypes are trained from a sequence of ..."
Abstract

Cited by 16 (8 self)
 Add to MetaCart
Learning vector quantization (LVQ) schemes constitute intuitive, powerful classification heuristics with numerous successful applications but, so far, limited theoretical background. We study LVQ rigorously within a simplifying model situation: two competing prototypes are trained from a sequence of examples drawn from a mixture of Gaussians. Concepts from statistical physics and the theory of online learning allow for an exact description of the training dynamics in highdimensional feature space. The analysis yields typical learning curves, convergence properties, and achievable generalization abilities. This is also possible for heuristic training schemes which do not relate to a cost function. We compare the performance of several algorithms, including Kohonen’s LVQ1 and LVQ+/, a limiting case of LVQ2.1. The former shows close to optimal performance, while LVQ+/ displays divergent behavior. We investigate how early stopping can overcome this difficulty. Furthermore, we study a crisp version of robust soft LVQ, which was recently derived from a statistical formulation. Surprisingly, it exhibits relatively poor generalization. Performance improves if a window for the selection of data is introduced; the resulting algorithm corresponds to cost function based LVQ2. The dependence of these results on the model parameters, for example, prior class probabilities, is investigated systematically, simulations confirm our analytical findings. Keywords: prototype based classification, learning vector quantization, WinnerTakesAll algorithms, online learning, competitive learning 1.
Improved learning of Riemannian metrics for exploratory analysis
, 2004
"... We have earlier introduced a principle for learning metrics, which shows how metricbased methods can be made to focus on discriminative properties of data. The main applications are in supervising unsupervised learning to model interesting variation in data, instead of modeling all variation as pl ..."
Abstract

Cited by 11 (4 self)
 Add to MetaCart
We have earlier introduced a principle for learning metrics, which shows how metricbased methods can be made to focus on discriminative properties of data. The main applications are in supervising unsupervised learning to model interesting variation in data, instead of modeling all variation as plain unsupervised learning does. The metrics are derived by approximations to an informationgeometric formulation. In this paper, we review the theory, introduce better approximations to the distances, and show how to apply them in two different kinds of unsupervised methods: prototypebased and pairwise distancebased. The two examples are selforganizing maps and multidimensional scaling (Sammon’s mapping).
Relevancebased feature extraction for hyperspectral images
 IEEE Trans. on Neural Networks
, 2008
"... Abstract—Hyperspectral imagery affords researchers all discriminating details needed for fine delineation of many material classes. This delineation is essential for scientific research ranging from geologic to environmental impact studies. In a data mining scenario, one cannot blindly discard infor ..."
Abstract

Cited by 9 (6 self)
 Add to MetaCart
Abstract—Hyperspectral imagery affords researchers all discriminating details needed for fine delineation of many material classes. This delineation is essential for scientific research ranging from geologic to environmental impact studies. In a data mining scenario, one cannot blindly discard information because it can destroy discovery potential. In a supervised classification scenario, however, the preselection of classes presents one with an opportunity to extract a reduced set of meaningful features without degrading classification performance. Given the complex correlations found in hyperspectral data and the potentially large number of classes, meaningful feature extraction is a difficult task. We turn to the recent neural paradigm of generalized relevance learning vector quantization (GRLVQ) [B. Hammer and T. Villmann, Neural Networks, vol. 15, pp. 1059–1068, 2002], which is based on, and substantially extends, learning vector
Distance learning in discriminative vector quantization
 Neural Computation
"... Discriminative vector quantization schemes such as learning vector quantization (LVQ) and extensions thereof offer efficient and intuitive classifiers which are based on the representation of classes by prototypes. The original methods, however, rely on the Euclidean distance corresponding to the as ..."
Abstract

Cited by 9 (6 self)
 Add to MetaCart
Discriminative vector quantization schemes such as learning vector quantization (LVQ) and extensions thereof offer efficient and intuitive classifiers which are based on the representation of classes by prototypes. The original methods, however, rely on the Euclidean distance corresponding to the assumption that the data can be represented by isotropic clusters. For this reason, extensions of the methods to more general metric structures have been proposed such as relevance adaptation in generalized LVQ (GLVQ) and matrix learning in GLVQ. In these approaches, metric parameters are learned based on the given classification task such that a data driven distance measure is found. In this article, we consider full matrix adaptation in advanced LVQ schemes; in particular, we introduce matrix learning to a recent statistical formalization of LVQ, robust soft LVQ, and we compare the results on several artificial and real life data sets to matrix learning in GLVQ, which is a derivation of LVQlike learning based on a (heuristic) cost function. In all cases, matrix adaptation allows a significant improvement of the classification accuracy. Interestingly, however, the principled behavior of the models with respect to prototype locations and extracted matrix dimensions shows several characteristic differences depending on the data sets.
Mathematical Aspects of Neural Networks
 European Symposium of Artificial Neural Networks 2003
, 2003
"... In this tutorial paper about mathematical aspects of neural networks, we will focus on two directions: on the one hand, we will motivate standard mathematical questions and well studied theory of classical neural models used in machine learning. On the other hand, we collect some recent theoretic ..."
Abstract

Cited by 6 (4 self)
 Add to MetaCart
In this tutorial paper about mathematical aspects of neural networks, we will focus on two directions: on the one hand, we will motivate standard mathematical questions and well studied theory of classical neural models used in machine learning. On the other hand, we collect some recent theoretical results (as of beginning of 2003) in the respective areas. Thereby, we follow the dichotomy offered by the overall network structure and restrict ourselves to feedforward networks, recurrent networks, and selforganizing neural systems, respectively.
Relevance LVQ versus SVM
 ARTIFICIAL INTELLIGENCE AND SOFTCOMPUTING, VOLUME 3070 OF SPRINGER LECTURE NOTES IN ARTIFICIAL INTELLIGENCE
, 2004
"... The support vector machine (SVM) constitutes one of the most successful current learning algorithms with excellent classification accuracy in large reallife problems and strong theoretical background. However, a SVM solution is given by a not intuitive classification in terms of extreme values o ..."
Abstract

Cited by 5 (3 self)
 Add to MetaCart
The support vector machine (SVM) constitutes one of the most successful current learning algorithms with excellent classification accuracy in large reallife problems and strong theoretical background. However, a SVM solution is given by a not intuitive classification in terms of extreme values of the training set and the size of a SVM classifier scales with the number of training data. Generalized
Incremental GRLVQ: Learning relevant features for 3D object recognition
 Neurocomputing
, 2008
"... We present a new variant of Generalized Learning Vector Quantization (GRLVQ) in a computer vision scenario. A version with incrementally added prototypes is used for the nontrivial case of highdimensional object recognition. Training is based upon a generic set of standard visual features, the lea ..."
Abstract

Cited by 4 (0 self)
 Add to MetaCart
We present a new variant of Generalized Learning Vector Quantization (GRLVQ) in a computer vision scenario. A version with incrementally added prototypes is used for the nontrivial case of highdimensional object recognition. Training is based upon a generic set of standard visual features, the learned input weights are used for iterative feature pruning. Thus, prototypes and input space are altered simultaneously, leading to very sparse and taskspecific representations. The effectiveness of the approach and the combination of the incremental variant together with pruning was tested on the Coil100 database. It exhibits excellent performance with regard to codebook size, feature selection and recognition accuracy. Key words: object recognition, relevance learning, feature selection, incremental learning vector quantization, adaptive metric 1
Generalized relevance learning vector quantization for classificationdriven feature extraction from hyperspectral data
 in Proc. American Society for Photogrammetry and Remote Sensing
, 2006
"... (GRLVQ) [1] is a “double action ” supervised neural learning machine that simultaneously adapts classification boundaries and a weighting of the input dimensions to reflect the relevance of each dimension for the given classification. It is thus a joint classification and feature extraction techniqu ..."
Abstract

Cited by 4 (3 self)
 Add to MetaCart
(GRLVQ) [1] is a “double action ” supervised neural learning machine that simultaneously adapts classification boundaries and a weighting of the input dimensions to reflect the relevance of each dimension for the given classification. It is thus a joint classification and feature extraction technique. In [2] we developed an improved version (GRLVQI) to handle intricate highdimensional data. GRLVQI makes significant headway of feature reduction for hyperspectral images without compromising classification accuracy. However, the number of features to which the data can be reduced in the original (reflectance data) domain is naturally limited by higher order correlations. Here we investigate GRLVQI processing on wavelet coefficients because of the approximately decorrelated nature and the sparsity of those coefficients. We investigate the DualTree Complex Wavelet Transform (DTCWT) [3] for its reduced oscillatory effects because spectral data often have discontinuities due to data fallout. We demonstrate that GRLVQI on the DTCWT coefficients achieves better classification with fewer features than using the Critically Sampled Discrete Wavelet Transform (CSDWT), which was already shown to yield better results with far fewer features than GRLVQI applied in the original data space. I.
A General Framework for SelfOrganizing Structure Processing Neural Networks
, 2003
"... Selforganization constitutes an important paradigm in machine learning with successful applications e.g. for data and webmining. However, so far most approaches have been proposed for data contained in a fixed and finite dimensional vector space. We will focus on extensions for more general dat ..."
Abstract

Cited by 4 (4 self)
 Add to MetaCart
Selforganization constitutes an important paradigm in machine learning with successful applications e.g. for data and webmining. However, so far most approaches have been proposed for data contained in a fixed and finite dimensional vector space. We will focus on extensions for more general data structures like sequences and tree structures in this article. Various extensions of the standard selforganizing map (SOM) to sequences or tree structures have been proposed in the literature: the temporal Kohonen map, the recursive SOM, and SOM for structured data (SOMSD), for example. These methods enhance the standard SOM by recursive connections. We define in this article a general recursive dynamic which enables the recursive processing of complex data structures based on recursively computed internal representations of the respective context. The above mechanisms of SOMs for structures are special cases of the proposed general dynamic, furthermore, the dynamic covers the supervised case of recurrent and recursive networks, too. The general framework offers a uniform notation for training mechanisms such as Hebbian learning and the transfer of alternatives such as vector quantization or the neural gas algorithm to structure processing networks. The formal definition of the recursive dynamic for structure processing unsupervised networks allows the transfer of theoretical issues from the SOM literature to the structure processing case. One can formulate general cost functions corresponding to vector quantization, neural gas, and a modification of SOM for the case of structures. The cost functions can be compared to Hebbian learning which can be interpreted as an approximation of a stochastic gradient descent. We derive as an alternative the exact gradien...