Results 1  10
of
41
The design and analysis of benchmark experiments
 J Comp Graph Stat
, 2005
"... The assessment of the performance of learners by means of benchmark experiments is an established exercise. In practice, benchmark studies are a tool to compare the performance of several competing algorithms for a certain learning problem. Crossvalidation or resampling techniques are commonly used ..."
Abstract

Cited by 29 (14 self)
 Add to MetaCart
(Show Context)
The assessment of the performance of learners by means of benchmark experiments is an established exercise. In practice, benchmark studies are a tool to compare the performance of several competing algorithms for a certain learning problem. Crossvalidation or resampling techniques are commonly used to derive point estimates of the performances which are compared to identify algorithms with good properties. For several benchmarking problems, test procedures taking the variability of those point estimates into account have been suggested. Most of the recently proposed inference procedures are based on special variance estimators for the crossvalidated performance. We introduce a theoretical framework for inference problems in benchmark experiments and show that standard statistical test procedures can be used to test for differences in the performances. The theory is based on well defined distributions of performance measures which can be compared with established tests. To demonstrate the usefulness in practice, the theoretical results are applied to regression and classification benchmark studies based on artificial and real world data.
Penalized loss functions for Bayesian model comparison
"... The deviance information criterion (DIC) is widely used for Bayesian model comparison, despite the lack of a clear theoretical foundation. DIC is shown to be an approximation to a penalized loss function based on the deviance, with a penalty derived from a crossvalidation argument. This approximati ..."
Abstract

Cited by 29 (2 self)
 Add to MetaCart
The deviance information criterion (DIC) is widely used for Bayesian model comparison, despite the lack of a clear theoretical foundation. DIC is shown to be an approximation to a penalized loss function based on the deviance, with a penalty derived from a crossvalidation argument. This approximation is valid only when the effective number of parameters in the model is much smaller than the number of independent observations. In disease mapping, a typical application of DIC, this assumption does not hold and DIC underpenalizes more complex models. Another deviancebased loss function, derived from the same decisiontheoretic framework, is applied to mixture models, which have previously been considered an unsuitable application for DIC.
Asymptotic equivalence of Bayes cross validation and widely applicable information criterion in singular learning theory
 Journal of Machine Learning Research
, 2010
"... In regular statistical models, the leaveoneout crossvalidation is asymptotically equivalent to the Akaike information criterion. However, since many learning machines are singular statistical models, the asymptotic behavior of the crossvalidation remains unknown. In previous studies, we establis ..."
Abstract

Cited by 16 (2 self)
 Add to MetaCart
In regular statistical models, the leaveoneout crossvalidation is asymptotically equivalent to the Akaike information criterion. However, since many learning machines are singular statistical models, the asymptotic behavior of the crossvalidation remains unknown. In previous studies, we established the singular learning theory and proposed a widely applicable information criterion, the expectation value of which is asymptotically equal to the average Bayes generalization loss. In the present paper, we theoretically compare the Bayes crossvalidation loss and the widely applicable information criterion and prove two theorems. First, the Bayes crossvalidation loss is asymptotically equivalent to the widely applicable information criterion as a random variable. Therefore, model selection and hyperparameter optimization using these two values are asymptotically equivalent. Second, the sum of the Bayes generalization error and the Bayes crossvalidation error is asymptotically equal to 2λ/n, where λ is the real log canonical threshold and n is the number of training samples. Therefore the relation between the crossvalidation error and the generalization error is determined by the algebraic geometrical structure of a learning machine. We also clarify that the deviance information criteria are different from the Bayes crossvalidation and the widely applicable information criterion.
Gaussian process regression with Studentt likelihood
"... In the Gaussian process regression the observation model is commonly assumed to be Gaussian, which is convenient in computational perspective. However, the drawback is that the predictive accuracy of the model can be significantly compromised if the observations are contaminated by outliers. A robus ..."
Abstract

Cited by 15 (4 self)
 Add to MetaCart
(Show Context)
In the Gaussian process regression the observation model is commonly assumed to be Gaussian, which is convenient in computational perspective. However, the drawback is that the predictive accuracy of the model can be significantly compromised if the observations are contaminated by outliers. A robust observation model, such as the Studentt distribution, reduces the influence of outlying observations and improves the predictions. The problem, however, is the analytically intractable inference. In this work, we discuss the properties of a Gaussian process regression model with the Studentt likelihood and utilize the Laplace approximation for approximate inference. We compare our approach to a variational approximation and a Markov chain Monte Carlo scheme, which utilize the commonly used scale mixture representation of the Studentt distribution. 1
Robust Gaussian Process Regression with a Studentt Likelihood
"... This paper considers the robust and efficient implementation of Gaussian process regression with a Studentt observation model, which has a nonlogconcave likelihood. The challenge with the Studentt model is the analytically intractable inference which is why several approximative methods have bee ..."
Abstract

Cited by 11 (3 self)
 Add to MetaCart
This paper considers the robust and efficient implementation of Gaussian process regression with a Studentt observation model, which has a nonlogconcave likelihood. The challenge with the Studentt model is the analytically intractable inference which is why several approximative methods have been proposed. Expectation propagation (EP) has been found to be a very accurate method in many empirical studies but the convergence of EP is known to be problematic with models containing nonlogconcave site functions. In this paper we illustrate the situations where standard EP fails to converge and review different modifications and alternative algorithms for improving the convergence. We demonstrate that convergence problems may occur during the typeII maximum a posteriori (MAP) estimation of the hyperparameters and show that standard EP may not converge in the MAP values with some difficult data sets. We present a robust implementation which relies primarily on parallel EP updates and uses a momentmatchingbased doubleloop algorithm with adaptively selected step size in difficult cases. The predictive performance of EP is compared with Laplace, variational Bayes, and Markov chain Monte Carlo approximations. Keywords: Gaussian process, robust regression, Studentt distribution, approximate inference, expectation propagation
Understanding predictive information criteria for Bayesian models ∗
, 2013
"... We review the Akaike, deviance, and WatanabeAkaike information criteria from a Bayesian perspective, where the goal is to estimate expected outofsampleprediction error using a biascorrectedadjustmentofwithinsampleerror. Wefocusonthechoicesinvolvedinsettingupthese measures, and we compare them i ..."
Abstract

Cited by 9 (2 self)
 Add to MetaCart
(Show Context)
We review the Akaike, deviance, and WatanabeAkaike information criteria from a Bayesian perspective, where the goal is to estimate expected outofsampleprediction error using a biascorrectedadjustmentofwithinsampleerror. Wefocusonthechoicesinvolvedinsettingupthese measures, and we compare them in three simple examples, one theoretical and two applied. The contribution of this paper is to put all these information criteria into a Bayesian predictive context and to better understand, through small examples, how these methods can apply in practice.
Bayesian Input Variable Selection Using Posterior Probabilities and Expected Utilities
, 2002
"... We consider the input variable selection in complex Bayesian hierarchical models. Our goal is to find a model with the smallest number of input variables having statistically or practically at least the same expected utility as the full model with all the available inputs. A good estimate for the ..."
Abstract

Cited by 8 (1 self)
 Add to MetaCart
(Show Context)
We consider the input variable selection in complex Bayesian hierarchical models. Our goal is to find a model with the smallest number of input variables having statistically or practically at least the same expected utility as the full model with all the available inputs. A good estimate for the expected utility can be computed using crossvalidation predictive densities. In the case of input selection and a large number of input combinations, the computation of the crossvalidation predictive densities for each model easily becomes computationally prohibitive. We propose to use the posterior probabilities obtained via variable dimension MCMC methods to find out potentially useful input combinations, for which the final model choice and assessment is done using the expected utilities.
Bayesian Modeling with Gaussian Processes using the GPstuff Toolbox,” arXiv:1206.5754 [cs, stat
, 2012
"... Gaussian processes (GP) are powerful tools for probabilistic modeling purposes. They can be used to define prior distributions over latent functions in hierarchical Bayesian models. The prior over functions is defined implicitly by the mean and covariance function, which determine the smoothness and ..."
Abstract

Cited by 8 (1 self)
 Add to MetaCart
Gaussian processes (GP) are powerful tools for probabilistic modeling purposes. They can be used to define prior distributions over latent functions in hierarchical Bayesian models. The prior over functions is defined implicitly by the mean and covariance function, which determine the smoothness and variability of the function. The inference can then be conducted directly in the function space by evaluating or approximating the posterior process. Despite their attractive theoretical properties GPs provide practical challenges in their implementation. GPstuff is a versatile collection of computational tools for GP models compatible with Linux and Windows MATLAB and Octave. It includes, among others, various inference methods, sparse approximations and tools for model assessment. In this work, we review these tools and demonstrate the use of GPstuff in several models. Last updated 20140411.
Neural Network Methods In Analysing And Modelling Time Varying Processes
, 2003
"... Teknillinen korkeakoulu Sähkö ja tietoliikennetekniikan osasto Laskennallisen tekniikan laboratorio Distribution: ..."
Abstract

Cited by 5 (0 self)
 Add to MetaCart
(Show Context)
Teknillinen korkeakoulu Sähkö ja tietoliikennetekniikan osasto Laskennallisen tekniikan laboratorio Distribution: