We present a class of algorithms for independent component analysis (ICA) which use contrast functions based on canonical correlations in a reproducing kernel Hilbert space. On the one hand, we show that our contrast functions are related to mutual information and have desirable mathematical properties as measures of statistical dependence. On the other hand, building on recent developments in kernel methods, we show that these criteria can be computed efficiently. Minimizing these criteria leads to flexible and robust algorithms for ICA. We illustrate with simulations involving a wide variety of source distributions, showing that our algorithms outperform many of the presently known algorithms. 1.

Introduction In blind source separation an N-channel sensor signal x(t) arises from M unknown scalar source signals s i (t), linearly mixed together by an unknown N M matrix A, and possibly corrupted by additive noise (t) x(t) = As(t) + (t) (1.1) We wish to estimate the mixing matrix A and the M-dimensional source signal s(t). Many natural signals can be sparsely represented in a proper signal dictionary s i (t) = K X k=1 C ik ' k (t) (1.2) The scalar functions ' k

This paper presents a new algorithm for the independent components analysis (ICA) problem based on an efficient entropy estimator. Like many previous methods, this algorithm directly minimizes the measure of departure from independence according to the estimated Kullback-Leibler divergence between the joint distribution and the product of the marginal distributions. We pair this approach with efficient entropy estimators from the statistics literature. In particular, the entropy estimator we use is consistent and exhibits rapid convergence. The algorithm based on this estimator is simple, computationally efficient, intuitively appealing, and outperforms other well known algorithms. In addition, the estimator's relative insensitivity to outliers translates into superior performance by our ICA algorithm on outlier tests. We present favorable comparisons to the Kernel ICA, FAST-ICA, JADE, and extended Infomax algorithms in extensive simulations. We also provide public domain source code for our algorithms.

Weinvestigate the use of linear and nonlinear principal manifolds for learning lowdimensional representations for visual recognition. Three techniques: Principal Component Analysis #PCA#, Independent Component Analysis #ICA# and Nonlinear PCA #NLPCA# are examined and tested in a visual recognition experiment using a large gallery of facial images from the #FERET" database. We compare the recognition performance of a nearest-neighbour matching rule with each principal manifold representation to that of a maximum aposteriori #MAP# matching rule using a Bayesian similarity measure derived from probabilistic subspaces and demonstrate the superiority of the latter.

We present a generalization of independent component analysis (ICA), where instead of looking for a linear transform that makes the data components independent, we look for a transform that makes the data components well fit by a tree-structured graphical model. This tree-dependent component analysis (TCA) provides a tractable and flexible approach to weakening the assumption of independence in ICA. In particular, TCA allows the underlying graph to have multiple connected components, and thus the method is able to find “clusters ” of components such that components are dependent within a cluster and independent between clusters. Finally, we make use of a notion of graphical models for time series due to Brillinger (1996) to extend these ideas to the temporal setting. In particular, we are able to fit models that incorporate tree-structured dependencies among multiple time series.

We applied independent component analysis (ICA) to hyperspectral images in order to learn an efficient representation of color in natural scenes. In the spectra of single pixels, the algorithm found basis functions that had broadband spectra and basis functions that were similar to natural reflectance spectra. When applied to small image patches, the algorithm found some basis functions that were achromatic and others with overall chromatic variation along lines in color space, indicating color opponency. The directions of opponency were not strictly orthogonal. Comparison with principal-component analysis on the basis of statistical measures such as average mutual information, kurtosis, and entropy, shows that the ICA transformation results in much sparser coefficients and gives higher coding efficiency. Our findings suggest that nonorthogonal opponent encoding of photoreceptor signals leads to higher coding efficiency and that ICA may be used to reveal the underlying statistical properties of color information in natural scenes.

A new algorithmic framework called denoising source separation (DSS) is introduced. The main benefit of this framework is that it allows for easy development of new source separation algorithms which are optimised for specific problems. In this framework, source separation algorithms are constucted around denoising procedures. The resulting algorithms can range from almost blind to highly specialised source separation algorithms. Both simple linear and more complex nonlinear or adaptive denoising schemes are considered. Some existing independent component analysis algorithms are reinterpreted within DSS framework and new, robust blind source separation algorithms are suggested. Although DSS algorithms need not be explicitly based on objective functions, there is often an implicit objective function that is optimised. The exact relation between the denoising procedure and the objective function is derived and a useful approximation of the objective function is presented. In the experimental section, various DSS schemes are applied extensively to artificial data, to real magnetoencephalograms and to simulated CDMA mobile network signals. Finally, various extensions to the proposed DSS algorithms are considered. These include nonlinear observation mappings, hierarchical models and overcomplete, nonorthogonal feature spaces. With these extensions, DSS appears to have relevance to many existing models of neural information processing.

Linear mixtures of independent random variables (the so-called sources) are sometimes referred to as under-determined mixtures (UDM) when the number of sources exceeds the dimension of the observation space. The algorithms proposed are able to identify algebraically a UDM using the second characteristic function (c.f.) of the observations, without any need of sparsity assumption on sources. In fact, by taking higher order derivatives of the multivariate c.f. core equation, the blind identification problem is shown to reduce to a tensor decomposition. With only two sensors, the first algorithm only needs a SVD. With a larger number of sensors, the second algorithm executes an alternating least squares (ALS) algorithm. The joint use of statistics of different orders is possible, and a LS solution can be computed. Identifiability conditions are stated in each of the two cases. Computer simulations eventually demonstrate performances in the absence of sparsity, and emphasize the interest in using jointly derivatives of different orders. r 2005 Elsevier B.V. All rights reserved.

It is often admitted that a static system with more inputs (sources) than outputs (sensors, or channels) cannot be blindly identified, that is, identified only from the observation of its outputs, and without any a priori knowledge on the source statistics but their independence. By resorting to High-Order Statistics, it turns out that static MIMO systems with fewer outputs than inputs can be identified, as demonstrated in the present paper. The principle, already described in a recent rather theoretical paper, had not yet been applied to a concrete blind identification problem. Here, in order to demonstrate its feasibility, the procedure is detailed in the case of a 2-sensor 3-source mixture; a numerical algorithm is devised, that blindly identifies a 3-input 2-output mixture. Computer results show its behavior as a function of the data length when sources are QPSK-modulated signals, widely used in digital communications. Then another algorithm is proposed to extract the 3 sources from the 2 observations, once the mixture has been identified. Contrary to the first algorithm, this one assumes that the sources have a known discrete distribution. Computer experiments are run in the case of three BPSK sources in presence of Gaussian noise.

In this paper we present a convolutive basis decomposition method and its application on simultaneous speakers separation from monophonic recordings. The model we propose is a convolutive version of the non-negative matrix factorization algorithm. Due to the non-negativity constraint this type of coding is very well suited for intuitively and efficiently representing magnitude spectra. We present results that reveal the nature of these basis functions and we introduce their utility in separating monophonic mixtures of known speakers.