Abstract

Abstract A new learning algorithm is derived which performs online stochastic gradient ascent in the mutual information between outputs and inputs of a network. In the absence of a priori knowledge about the `signal ' and `noise ' components of the input, propagation of information depends on calibrating network non-linearities to the detailed higher-order moments of the input density functions. By incidentally minimising mutual information between outputs, as well as maximising their individual entropies, the network `factorises ' the input into independent components. As an example application, we have achieved near-perfect separation of ten digitally mixed speech signals. Our simulations lead us to believe that our network performs better at blind separation than the HeraultJutten network, reflecting the fact that it is derived rigorously from the mutual information objective. 1 Introduction Unsupervised learning algorithms based on information theoretic principles have tended to focus on linear decorrelation (Barlow & F"oldi'ak 1989) or maximisation of signal-to-noise ratios assuming Gaussian sources (Linsker 1992). With the exception of (Becker 1992), there has been little attempt to use non-linearity in networks to achieve something a linear network could not. Non-linear networks, however, are capable of computing more general statistics than those second-order ones involved in decorrelation, and as a consequence they are capable of dealing with signals (and noises) which have detailed higher-order structure. The success of the `H-J ' networks at blind separation (Jutten & Herault 1991) suggests that it should be possible to separate statistically independent components, by using learning rules which make use of moments of all orders.

Citations