Results 1  10
of
19
Advances in nonlinear blind source separation
 In Proc. of the 4th Int. Symp. on Independent Component Analysis and Blind Signal Separation (ICA2003
, 2003
"... Abstract — In this paper, we briefly review recent advances in blind source separation (BSS) for nonlinear mixing models. After a general introduction to the nonlinear BSS and ICA (independent Component Analysis) problems, we discuss in more detail uniqueness issues, presenting some new results. A f ..."
Abstract

Cited by 30 (2 self)
 Add to MetaCart
Abstract — In this paper, we briefly review recent advances in blind source separation (BSS) for nonlinear mixing models. After a general introduction to the nonlinear BSS and ICA (independent Component Analysis) problems, we discuss in more detail uniqueness issues, presenting some new results. A fundamental difficulty in the nonlinear BSS problem and even more so in the nonlinear ICA problem is that they are nonunique without extra constraints, which are often implemented by using a suitable regularization. Postnonlinear mixtures are an important special case, where a nonlinearity is applied to linear mixtures. For such mixtures, the ambiguities are essentially the same as for the linear ICA or BSS problems. In the later part of this paper, various separation techniques proposed for postnonlinear mixtures and general nonlinear mixtures are reviewed. I. THE NONLINEAR ICA AND BSS PROBLEMS Consider Æ samples of the observed data vector Ü, modeled by
Variational learning and bitsback coding: an informationtheoretic view to Bayesian learning
 IEEE Transactions on Neural Networks
"... Abstract—The bitsback coding first introduced by Wallace in 1990 and later by Hinton and van Camp in 1993 provides an interesting link between Bayesian learning and informationtheoretic minimumdescriptionlength (MDL) learning approaches. The bitsback coding allows interpreting the cost function ..."
Abstract

Cited by 17 (7 self)
 Add to MetaCart
Abstract—The bitsback coding first introduced by Wallace in 1990 and later by Hinton and van Camp in 1993 provides an interesting link between Bayesian learning and informationtheoretic minimumdescriptionlength (MDL) learning approaches. The bitsback coding allows interpreting the cost function used in the variational Bayesian method called ensemble learning as a code length in addition to the Bayesian view of misfit of the posterior approximation and a lower bound of model evidence. Combining these two viewpoints provides interesting insights to the learning process and the functions of different parts of the model. In this paper, the problem of variational Bayesian learning of hierarchical latent variable models is used to demonstrate the benefits of the two views. The codelength interpretation provides new views to many parts of the problem such as model comparison and pruning and helps explain many phenomena occurring in learning. Index Terms—Bitsback coding, ensemble learning, hierarchical latent variable models, minimum description length, variational Bayesian learning. I.
Building Blocks For Variational Bayesian Learning Of Latent Variable Models
 JOURNAL OF MACHINE LEARNING RESEARCH
, 2006
"... We introduce standardised building blocks designed to be used with variational Bayesian learning. The blocks include Gaussian variables, summation, multiplication, nonlinearity, and delay. A large variety of latent variable models can be constructed from these blocks, including variance models a ..."
Abstract

Cited by 11 (8 self)
 Add to MetaCart
We introduce standardised building blocks designed to be used with variational Bayesian learning. The blocks include Gaussian variables, summation, multiplication, nonlinearity, and delay. A large variety of latent variable models can be constructed from these blocks, including variance models and nonlinear modelling, which are lacking from most existing variational systems. The introduced blocks are designed to fit together and to yield e#cient update rules. Practical implementation of various models is easy thanks to an associated software package which derives the learning formulas automatically once a specific model structure has been fixed. Variational Bayesian learning provides a cost function which is used both for updating the variables of the model and for optimising the model structure. All the computations can be carried out locally, resulting in linear computational complexity. We present
Estimating Functions for Blind Separation When Sources Have VarianceDependencies
, 2004
"... The blind separation problem where the sources are not independent, but have variancedependencies is discussed. Hyvärinen and Hurri[1] proposed an algorithm which requires no assumption on distributions of sources and no parametric model of dependencies between components. In this paper, we exte ..."
Abstract

Cited by 9 (2 self)
 Add to MetaCart
The blind separation problem where the sources are not independent, but have variancedependencies is discussed. Hyvärinen and Hurri[1] proposed an algorithm which requires no assumption on distributions of sources and no parametric model of dependencies between components. In this paper, we extend the semiparametric statistical approach of Amari and Cardoso[2] under variancedependencies and study estimating functions for blind separation of such dependent sources. In particular, we show that many of ICA algorithms are applicable to the variancedependent model as well. Our theoretical consequences were confirmed by artificial and realistic examples.
Partially observed values, in
 Proc. Int. Joint Conf. on Neural Networks (IJCNN 2004
, 2004
"... It is common to have both observed and missing values in data. This paper concentrates on the case where a value can be somewhere between those two ends, partially observed and partially missing. To achieve that, a method of using evidence nodes in a Bayesian network is studied. Different ways of ha ..."
Abstract

Cited by 7 (4 self)
 Add to MetaCart
It is common to have both observed and missing values in data. This paper concentrates on the case where a value can be somewhere between those two ends, partially observed and partially missing. To achieve that, a method of using evidence nodes in a Bayesian network is studied. Different ways of handling inaccuracies are discussed in examples and the proposed approach is justified in the experiments with real image data.
Bayes Blocks: An implementation of the variational Bayesian building blocks framework
 In Proceedings of the 21st Conference on Uncertainty in Artificial Intelligence, UAI 2005
, 2005
"... A software library for constructing and learning probabilistic models is presented. The library offers a set of building blocks from which a large variety of static and dynamic models can be built. These include hierarchical models for variances of other variables and many nonlinear models. The unde ..."
Abstract

Cited by 7 (5 self)
 Add to MetaCart
A software library for constructing and learning probabilistic models is presented. The library offers a set of building blocks from which a large variety of static and dynamic models can be built. These include hierarchical models for variances of other variables and many nonlinear models. The underlying variational Bayesian machinery, providing for fast and robust estimation but being mathematically rather involved, is almost completely hidden from the user thus making it very easy to use the library. The building blocks include Gaussian, rectified Gaussian and mixtureofGaussians variables and computational nodes which can be combined rather freely. 1
Temporal and Spatiotemporal Coherence in SimpleCell Responses: A Generative . . .
, 2003
"... We present a twolayer dynamic generative model of the statistical structure of natural image sequences. The second layer of the model is a linear mapping from simplecell outputs to pixel values, as in most work on natural image statistics. The first layer models the dependencies of the activity le ..."
Abstract

Cited by 6 (4 self)
 Add to MetaCart
We present a twolayer dynamic generative model of the statistical structure of natural image sequences. The second layer of the model is a linear mapping from simplecell outputs to pixel values, as in most work on natural image statistics. The first layer models the dependencies of the activity levels (amplitudes or variances) of the simple cells, using a multivariate autoregressive model. The second layer shows the emergence of basis vectors that are localized, oriented and have different scales, just like in previous work. But in our new model, the first layer learns connections between the simple cells that are similar to complex cell pooling: connections are strong among cells with similar preferred location, frequency and orientation. In contrast to previous work in which one of the layers needed to be fixed in advance, the dynamic model enables us to estimate both of the layers simultaneously fromnatural data.
Variational Bayes for continuoustime nonlinear statespace models
 In NIPS*2006 Workshop on Dynamical Systems, Stochastic Processes and Bayesian Inference
, 2006
"... We present an extension of the variational Bayesian nonlinear statespace model introduced by Valpola and Karhunen in 2002 [1] for continuoustime models. The model is based on using multilayer perceptron (MLP) networks to model the nonlinearities. Moving to continuoustime requires solving a stocha ..."
Abstract

Cited by 5 (4 self)
 Add to MetaCart
We present an extension of the variational Bayesian nonlinear statespace model introduced by Valpola and Karhunen in 2002 [1] for continuoustime models. The model is based on using multilayer perceptron (MLP) networks to model the nonlinearities. Moving to continuoustime requires solving a stochastic differential equation (SDE) to evaluate the predictive distribution of the states, but otherwise all computation happens as in the discretetime case. The close connection between the methods allows utilising our new improved state inference method for both discretetime and continuoustime modelling. 1
Online variational Bayesian learning
 In Proc. of the 4th Int. Symp. on Independent Component Analysis and Blind Signal Separation (ICA2003
, 2003
"... Variational Bayesian learning is an approximation to the exact Bayesian learning where the true posterior is approximated with a simpler distribution. In this paper we present an online variant of variational Bayesian learning. The method is based on collecting likelihood information as the trainin ..."
Abstract

Cited by 4 (0 self)
 Add to MetaCart
Variational Bayesian learning is an approximation to the exact Bayesian learning where the true posterior is approximated with a simpler distribution. In this paper we present an online variant of variational Bayesian learning. The method is based on collecting likelihood information as the training samples are processed one at a time and decaying the old likelihood information. The decay or forgetting is very important since otherwise the system would get stuck to the first reasonable solution it finds. The method is tested with a simple linear independent component analysis (ICA) problem but it can easily be applied to other more difficult problems. 1.
Approximate riemannian conjugate gradient learning for fixedform variational bayes
 Journal of Machine Learning Research
"... Variational Bayesian (VB) methods are typically only applied to models in the conjugateexponential family using the variational Bayesian expectation maximisation (VB EM) algorithm or one of its variants. In this paper we present an efficient algorithm for applying VB to more general models. The met ..."
Abstract

Cited by 4 (1 self)
 Add to MetaCart
Variational Bayesian (VB) methods are typically only applied to models in the conjugateexponential family using the variational Bayesian expectation maximisation (VB EM) algorithm or one of its variants. In this paper we present an efficient algorithm for applying VB to more general models. The method is based on specifying the functional form of the approximation, such as multivariate Gaussian. The parameters of the approximation are optimised using a conjugate gradient algorithm that utilises the Riemannian geometry of the space of the approximations. This leads to a very efficient algorithm for suitably structured approximations. It is shown empirically that the proposed method is comparable or superior in efficiency to the VB EM in a case where both are applicable. We also apply the algorithm to learning a nonlinear statespace model and a nonlinear factor analysis model for which the VB EM is not applicable. For these models, the proposed algorithm outperforms alternative gradientbased methods by a significant margin.