Results 1 
8 of
8
A framework for evaluating approximation methods for Gaussian process regression
 Journal of Machine Learning Research
"... Gaussian process (GP) predictors are an important component of many Bayesian approaches to machine learning. However, even a straightforward implementation of Gaussian process regression (GPR) requires O(n2) space and O(n3) time for a dataset of n examples. Several approximation methods have been pr ..."
Abstract

Cited by 17 (0 self)
 Add to MetaCart
Gaussian process (GP) predictors are an important component of many Bayesian approaches to machine learning. However, even a straightforward implementation of Gaussian process regression (GPR) requires O(n2) space and O(n3) time for a dataset of n examples. Several approximation methods have been proposed, but there is a lack of understanding of the relative merits of the different approximations, and in what situations they are most useful. We recommend assessing the quality of the predictions obtained as a function of the compute time taken, and comparing to standard baselines (e.g., Subset of Data and FITC). We empirically investigate four different approximation algorithms on four different prediction problems, and make our code available to encourage future comparisons.
1 Scaling Multidimensional Inference for Structured Gaussian Processes
, 1209
"... Abstract—Exact Gaussian Process (GP) regression has O(N 3) runtime for data size N, making it intractable for large N. Many algorithms for improving GP scaling approximate the covariance with lower rank matrices. Other work has exploited structure inherent in particular covariance functions, includi ..."
Abstract

Cited by 6 (1 self)
 Add to MetaCart
(Show Context)
Abstract—Exact Gaussian Process (GP) regression has O(N 3) runtime for data size N, making it intractable for large N. Many algorithms for improving GP scaling approximate the covariance with lower rank matrices. Other work has exploited structure inherent in particular covariance functions, including GPs with implied Markov structure, and equispaced inputs (both enable O(N) runtime). However, these GP advances have not been extended to the multidimensional input setting, despite the preponderance of multidimensional applications. This paper introduces and tests novel extensions of structured GPs to multidimensional inputs. We present new methods for additive GPs, showing a novel connection between the classic backfitting method and the Bayesian framework. To achieve optimal accuracycomplexity tradeoff, we extend this model with a novel variant of projection pursuit regression. Our primary result – projection pursuit Gaussian Process Regression – shows orders of magnitude speedup while preserving high accuracy. The natural second and third steps include nonGaussian observations and higher dimensional equispaced grid methods. We introduce novel techniques to address both of these necessary directions. We thoroughly illustrate the power of these three advances on several datasets, achieving close performance to the naive Full GP at orders of magnitude less cost. Index Terms—Gaussian Processes, Backfitting, ProjectionPursuit Regression, Kronecker matrices. 1
Hilbert space methods for reducedrank Gaussian process regression. arXiv preprint 1401.5508
, 2014
"... This paper proposes a novel scheme for reducedrank Gaussian process regression. The method is based on an approximate series expansion of the covariance function in terms of an eigenfunction expansion of the Laplace operator in a compact subset of Rd. On this approximate eigenbasis the eigenvalues ..."
Abstract

Cited by 3 (0 self)
 Add to MetaCart
This paper proposes a novel scheme for reducedrank Gaussian process regression. The method is based on an approximate series expansion of the covariance function in terms of an eigenfunction expansion of the Laplace operator in a compact subset of Rd. On this approximate eigenbasis the eigenvalues of the covariance function can be expressed as simple functions of the spectral density of the Gaussian process, which allows the GP inference to be solved under a computational cost scaling as O(nm2) (initial) and O(m3) (hyperparameter learning) with m basis functions and n data points. The approach also allows for rigorous error analysis with Hilbert space theory, and we show that the approximation becomes exact when the size of the compact subset and the number of eigenfunctions go to infinity. The expansion generalizes to Hilbert spaces with an inner product which is defined as an integral over a specified input density. The method is compared to previously proposed methods theoretically and through empirical tests with simulated and real data.
Efficient Gaussian process inference for shortscale spatiotemporal modeling. In:
 Proceedings of the 15th International Conference on Artificial Intelligence and Statistics.
, 2012
"... Abstract This paper presents an efficient Gaussian process inference scheme for modeling shortscale phenomena in spatiotemporal datasets. Our model uses a sum of separable, compactly supported covariance functions, which yields a full covariance matrix represented in terms of small sparse matrices ..."
Abstract

Cited by 2 (0 self)
 Add to MetaCart
(Show Context)
Abstract This paper presents an efficient Gaussian process inference scheme for modeling shortscale phenomena in spatiotemporal datasets. Our model uses a sum of separable, compactly supported covariance functions, which yields a full covariance matrix represented in terms of small sparse matrices operating either on the spatial or temporal domain. The proposed inference procedure is based on Gibbs sampling, in which samples from the conditional distribution of the latent function values are obtained by applying a simple linear transformation to samples drawn from the joint distribution of the function values and the observations. We make use of the proposed model structure and the conjugate gradient method to compute the required transformation. In the experimental part, the proposed algorithm is compared to the standard approach using the sparse Cholesky decomposition and it is shown to be much faster and computationally feasible for 1001000 times larger datasets. We demonstrate the advantages of the proposed method in the problem of reconstructing sea surface temperature, which requires processing of a realworld dataset with 10 6 observations.
aan de Technische Universiteit Delft;
"... ter verkrijging van de graad van doctor ..."
(Show Context)
A Framework for Evaluating Approximation Methods for Gaussian Process Regression
"... Gaussian process (GP) predictors are an important component of many Bayesian approaches to machine learning. However, even a straightforward implementation of Gaussian process regression (GPR) requires O(n 2) space and O(n 3) time for a data set of n examples. Several approximation methods have been ..."
Abstract
 Add to MetaCart
Gaussian process (GP) predictors are an important component of many Bayesian approaches to machine learning. However, even a straightforward implementation of Gaussian process regression (GPR) requires O(n 2) space and O(n 3) time for a data set of n examples. Several approximation methods have been proposed, but there is a lack of understanding of the relative merits of the different approximations, and in what situations they are most useful. We recommend assessing the quality of the predictions obtained as a function of the compute time taken, and comparing to standard baselines (e.g., Subset of Data and FITC). We empirically investigate four different approximation algorithms on four different prediction problems, and make our code available to encourage future comparisons.
ON CONVERGENCE AND ACCURACY OF STATESPACE APPROXIMATIONS OF SQUARED EXPONENTIAL COVARIANCE FUNCTIONS
"... In this paper we study the accuracy and convergence of statespace approximations of Gaussian processes (GPs) with squared exponential (SE) covariance functions. This kind of approximations is important in construction of Kalman filtering and smoothing based GP regression algorithms, which have a ..."
Abstract
 Add to MetaCart
In this paper we study the accuracy and convergence of statespace approximations of Gaussian processes (GPs) with squared exponential (SE) covariance functions. This kind of approximations is important in construction of Kalman filtering and smoothing based GP regression algorithms, which have a linear (as opposed to conventional cubic) computational complexity in the number of training samples. We start by deriving general conditions for a spectral density approximation to give a uniform convergence of the mean and covariance functions. We then show that the previously proposed reciprocal Taylor series approximation gives such uniform convergence. We then derive new approximations based on Pade ́ approximants of the exponential function as well as approximations inspired by the central limit theorem, and prove their uniform convergence. Finally, we compare accuracy of the different approximations numerically. Index Terms — Gaussian process regression, statespace approximation, squared exponential, Kalman filter and smoother, Padé