Results 1  10
of
78
Independent Component Analysis
 Neural Computing Surveys
, 2001
"... A common problem encountered in such disciplines as statistics, data analysis, signal processing, and neural network research, is nding a suitable representation of multivariate data. For computational and conceptual simplicity, such a representation is often sought as a linear transformation of the ..."
Abstract

Cited by 1492 (93 self)
 Add to MetaCart
A common problem encountered in such disciplines as statistics, data analysis, signal processing, and neural network research, is nding a suitable representation of multivariate data. For computational and conceptual simplicity, such a representation is often sought as a linear transformation of the original data. Wellknown linear transformation methods include, for example, principal component analysis, factor analysis, and projection pursuit. A recently developed linear transformation method is independent component analysis (ICA), in which the desired representation is the one that minimizes the statistical dependence of the components of the representation. Such a representation seems to capture the essential structure of the data in many applications. In this paper, we survey the existing theory and methods for ICA. 1
Optimal Unsupervised Learning in a SingleLayer Linear Feedforward Neural Network
, 1989
"... A new approach to unsupervised learning in a singlelayer linear feedforward neural network is discussed. An optimality principle is proposed which is based upon preserving maximal information in the output units. An algorithm for unsupervised learning based upon a Hebbian learning rule, which achie ..."
Abstract

Cited by 218 (0 self)
 Add to MetaCart
A new approach to unsupervised learning in a singlelayer linear feedforward neural network is discussed. An optimality principle is proposed which is based upon preserving maximal information in the output units. An algorithm for unsupervised learning based upon a Hebbian learning rule, which achieves the desired optimality is presented, The algorithm finds the eigenvectors of the input correlation matrix, and it is proven to converge with probability one. An implementation which can train neural networks using only local "synaptic" modification rules is described. It is shown that the algorithm is closely related to algorithms in statistics (Factor Analysis and Principal Components Analysis) and neural networks (Selfsupervised Backpropagation, or the "encoder" problem). It thus provides an explanation of certain neural network behavior in terms of classical statistical techniques. Examples of the use of a linear network for solving image coding and texture segmentation problems are presented. Also, it is shown that the algorithm can be used to find "visual receptive fields" which are qualitatively similar to those found in primate retina and visual cortex.
Adaptive Network for Optimal Linear Feature Extraction
"... A network of highly interconnected linear neuronlike processing units and a simple, local, unsupcrvised rule for the modification of connection strengths between these units are proposed. After training the network on a high (m) dimensional distribution of input vectors, the lower (n) dimensional o ..."
Abstract

Cited by 63 (0 self)
 Add to MetaCart
A network of highly interconnected linear neuronlike processing units and a simple, local, unsupcrvised rule for the modification of connection strengths between these units are proposed. After training the network on a high (m) dimensional distribution of input vectors, the lower (n) dimensional output will be a projection into the subspace of the n largest principal components (the subspace spanned by the n eigenvectors of largest eigenvalues of the input covariance matrix) and maximize the mutual information between the input and the output in the same way as principal component analysis does. The purely local natu of the synaptic modification rule (simple Hebbian and antiHebbian) makes the implementation of the network easier, faster and biologically more plausible than rules depending on error propagation.
Conditions for nonnegative independent component analysis
 IEEE Signal Processing Letters
, 2002
"... We consider the noiseless linear independent component analysis problem, in the case where the hidden sources s are nonnegative. We assume that the random variables s i s are wellgrounded in that they have a nonvanishing pdf in the (positive) neighbourhood of zero. For an orthonormal rotation y = ..."
Abstract

Cited by 63 (11 self)
 Add to MetaCart
We consider the noiseless linear independent component analysis problem, in the case where the hidden sources s are nonnegative. We assume that the random variables s i s are wellgrounded in that they have a nonvanishing pdf in the (positive) neighbourhood of zero. For an orthonormal rotation y = Wx of prewhitened observations x = QAs, under certain reasonable conditions we show that y is a permutation of the s (apart from a scaling factor) if and only if y is nonnegative with probability 1. We suggest that this may enable the construction of practical learning algorithms, particularly for sparse nonnegative sources.
Learning in Linear Neural Networks: a Survey
 IEEE Transactions on neural networks
, 1995
"... Networks of linear units are the simplest kind of networks, where the basic questions related to learning, generalization, and selforganisation can sometimes be answered analytically. We survey most of the known results on linear networks, including: (1) backpropagation learning and the structure ..."
Abstract

Cited by 56 (4 self)
 Add to MetaCart
Networks of linear units are the simplest kind of networks, where the basic questions related to learning, generalization, and selforganisation can sometimes be answered analytically. We survey most of the known results on linear networks, including: (1) backpropagation learning and the structure of the error function landscape; (2) the temporal evolution of generalization; (3) unsupervised learning algorithms and their properties. The connections to classical statistical ideas, such as principal component analysis (PCA), are emphasized as well as several simple but challenging open questions. A few new results are also spread across the paper, including an analysis of the effect of noise on backpropagation networks and a unified view of all unsupervised algorithms. Keywords linear networks, supervised and unsupervised learning, Hebbian learning, principal components, generalization, local minima, selforganisation I. Introduction This paper addresses the problems of supervise...
Candid covariancefree incremental principal component analysis
 IEEE Trans. Pattern Analysis and Machine Intelligence
, 2003
"... Abstract—Appearancebased image analysis techniques require fast computation of principal components of highdimensional image vectors. We introduce a fast incremental principal component analysis (IPCA) algorithm, called candid covariancefree IPCA (CCIPCA), used to compute the principal components ..."
Abstract

Cited by 56 (9 self)
 Add to MetaCart
Abstract—Appearancebased image analysis techniques require fast computation of principal components of highdimensional image vectors. We introduce a fast incremental principal component analysis (IPCA) algorithm, called candid covariancefree IPCA (CCIPCA), used to compute the principal components of a sequence of samples incrementally without estimating the covariance matrix (so covariancefree). The new method is motivated by the concept of statistical efficiency (the estimate has the smallest variance given the observed data). To do this, it keeps the scale of observations and computes the mean of observations incrementally, which is an efficient estimate for some wellknown distributions (e.g., Gaussian), although the highest possible efficiency is not guaranteed in our case because of unknown sample distribution. The method is for realtime applications and, thus, it does not allow iterations. It converges very fast for highdimensional image vectors. Some links between IPCA and the development of the cerebral cortex are also discussed. Index Terms—Principal component analysis, incremental principal component analysis, stochastic gradient ascent (SGA), generalized hebbian algorithm (GHA), orthogonal complement. æ 1
Independent Component Analysis by General Nonlinear Hebbianlike Learning Rules
 Signal Processing
, 1998
"... A number of neural learning rules have been recently proposed... In this paper, we show that in fact, ICA can be performed by very simple Hebbian or antiHebbian learning rules, which may have only weak relations to such informationtheoretical quantities. Rather suprisingly, practically any nonlin ..."
Abstract

Cited by 56 (11 self)
 Add to MetaCart
A number of neural learning rules have been recently proposed... In this paper, we show that in fact, ICA can be performed by very simple Hebbian or antiHebbian learning rules, which may have only weak relations to such informationtheoretical quantities. Rather suprisingly, practically any nonlinear function can be used in the learning rule, provided only that the sign of the Hebbian/antiHebbian term is chosen correctly. In addition to the Hebbianlike mechanism, the weight vector is here constrained to have unit norm, and the data is preprocessed by prewhitening, or sphering. These results imply that one can choose the nonlinearity so as to optimize desired statistical or numerical criteria.
Neural network approaches to image compression
 Proc. IEEE
, 1995
"... Abstract — This paper presents a tutorial overview of neural networks as signal processing tools for image compression. They are well suited to the problem of image compression due to their massively parallel and distributed architecture. Their characteristics are analogous to some of the features o ..."
Abstract

Cited by 34 (1 self)
 Add to MetaCart
Abstract — This paper presents a tutorial overview of neural networks as signal processing tools for image compression. They are well suited to the problem of image compression due to their massively parallel and distributed architecture. Their characteristics are analogous to some of the features of our own visual system, which allow us to process visual information with much ease. For example, multilayer perceptrons can be used as nonlinear predictors in differential pulsecode modulation (DPCM). Such predictors have been shown to increase the predictive gain relative to a linear predictor. Another active area of research is in the application of Hebbian learning to the extraction of principal components, which are the basis vectors for the optimal linear KarhunenLoève transform (KLT). These learning algorithms are iterative, have some computational advantages over standard eigendecomposition techniques, and can be made to adapt to changes in the input signal. Yet another model, the selforganizing feature map (SOFM), has been used with a great deal of success in the design of codebooks for vector quantization (VQ). The resulting codebooks are less sensitive to initial conditions than the standard LBG algorithm, and the topological ordering of the entries can be exploited to further increase coding efficiency and reduce computational complexity. I.
A ‘nonnegative PCA’ algorithm for independent component analysis, 2002, submitted for publication
"... We consider the task of independent component analysis when the independent sources are known to be nonnegative and wellgrounded, so that they have a nonzero probability density function (pdf) in the region of zero. We propose the use of a "nonnegative principal component analysis (nonnegative PCA) ..."
Abstract

Cited by 26 (2 self)
 Add to MetaCart
We consider the task of independent component analysis when the independent sources are known to be nonnegative and wellgrounded, so that they have a nonzero probability density function (pdf) in the region of zero. We propose the use of a "nonnegative principal component analysis (nonnegative PCA) " algorithm, which is a special case of the nonlinear PCA algorithm, but with a rectification nonlinearity, and we conjecture that this algorithm will find such nonnegative wellgrounded independent sources, under reasonable initial conditions. While the algorithm has proved difficult to analyze in the general case, we give some analytical results that are consistent with this conjecture and some numerical simulations that illustrate its operation. Index Terms independent component analysis learning (artificial intelligence) matrix decomposition principal component analysis independent component analysis nonlinear principal component analysis nonnegative PCA algorithm nonnegative matrix factorization nonzero probability density function rectification nonlinearity subspace learning rule ©2004 IEEE. Personal use of this material is permitted. However, permission to reprint/republish this material for advertising or promotional purposes or for creating new collective works for resale or redistribution to servers or lists, or to reuse any
General Results on the Convergence of Stochastic Algorithms
 IEEE Transactions on Automatic Control
, 1996
"... A deterministic approach is proposed for proving the convergence of stochastic algorithms of the most general form, under necessary conditions on the input noise, and reasonable conditions on the (nonnecessarily continuous) mean field. Emphasis is made on the case where more than one stationary poi ..."
Abstract

Cited by 25 (0 self)
 Add to MetaCart
A deterministic approach is proposed for proving the convergence of stochastic algorithms of the most general form, under necessary conditions on the input noise, and reasonable conditions on the (nonnecessarily continuous) mean field. Emphasis is made on the case where more than one stationary point exist. We use also this approach to prove the convergence of stochastic algorithm with Markovian dynamics. 1 Introduction The general structure of stochastic algorithms is the following : ` n = ` n\Gamma1 + fl n H(`n\Gamma1 ; Xn ) (1) fl n is a nonnegative decreasing sequence, typically 1=n (or 1=n 2=3 when an averaging technique is used, cf [18]), Xn is a "somehow stationary" sequence and ` n is at step n the estimated solution of E[H(`; X)] = 0 where the expectation is taken over the distribution of X. Stochastic algorithms have a wide range of application in recursive system identification, adaptive filtering, pattern recognition, adaptive learning [20], sequential change detec...