Results 11  20
of
62
Bayesian Model Assessment and Comparison Using CrossValidation Predictive Densities
 Neural Computation
, 2002
"... In this work, we discuss practical methods for the assessment, comparison, and selection of complex hierarchical Bayesian models. A natural way to assess the goodness of the model is to estimate its future predictive capability by estimating expected utilities. Instead of just making a point estimat ..."
Abstract

Cited by 26 (10 self)
 Add to MetaCart
In this work, we discuss practical methods for the assessment, comparison, and selection of complex hierarchical Bayesian models. A natural way to assess the goodness of the model is to estimate its future predictive capability by estimating expected utilities. Instead of just making a point estimate, it is important to obtain the distribution of the expected utility estimate, as it describes the uncertainty in the estimate. The distributions of the expected utility estimates can also be used to compare models, for example, by computing the probability of one model having a better expected utility than some other model. We propose an approach using crossvalidation predictive densities to obtain expected utility estimates and Bayesian bootstrap to obtain samples from their distributions. We also discuss the probabilistic assumptions made and properties of two practical crossvalidation methods, importance sampling and kfold crossvalidation. As illustrative examples, we use MLP neural networks and Gaussian Processes (GP) with Markov chain Monte Carlo sampling in one toy problem and two challenging realworld problems.
Models and Selection Criteria for Regression and Classification
 Uncertainty in Arificial Intelligence 13
, 1997
"... When performing regression or classification, we are interested in the conditional probability distribution for an outcome or class variable Y given a set of explanatory or input variables X. We consider Bayesian models for this task. In particular, we examine a special class of models, which we ca ..."
Abstract

Cited by 23 (2 self)
 Add to MetaCart
When performing regression or classification, we are interested in the conditional probability distribution for an outcome or class variable Y given a set of explanatory or input variables X. We consider Bayesian models for this task. In particular, we examine a special class of models, which we call Bayesian regression/classification (BRC) models, that can be factored into independent conditional (yjx) and input (x) models. These models are convenient, because the conditional model (the portion of the full model that we care about) can be analyzed by itself. We examine the practice of transforming arbitrary Bayesian models to BRC models, and argue that this practice is often inappropriate because it ignores prior knowledge that may be important for learning. In addition, we examine Bayesian methods for learning models from data. We discuss two criteria for Bayesian model selection that are appropriate for repression/classification: one described by Spiegelhalter et al. (1993), and an...
A Comparison of Scientific and Engineering Criteria for Bayesian Model Selection
 Statistics and Computing
, 1996
"... this paper, we assume that there are a finite number of possible true models. For each possible model m, we define the random (vector) variable \Theta m whose values correspond to the possible values of the parameters for m. We encode our uncertainty about \Theta m using the probability distribution ..."
Abstract

Cited by 19 (0 self)
 Add to MetaCart
this paper, we assume that there are a finite number of possible true models. For each possible model m, we define the random (vector) variable \Theta m whose values correspond to the possible values of the parameters for m. We encode our uncertainty about \Theta m using the probability distribution p(\Theta m jm). In this paper, we assume that p(\Theta m jm) is a probability density function. Given random sample D, we compute the posterior distributions for M and each \Theta m
Calibrated probabilistic forecasting at the Stateline wind energy center: The regimeswitching spacetime (RST) method
 Journal of the American Statistical Association
, 2004
"... With the global proliferation of wind power, accurate shortterm forecasts of wind resources at wind energy sites are becoming paramount. Regimeswitching spacetime (RST) models merge meteorological and statistical expertise to obtain accurate and calibrated, fully probabilistic forecasts of wind s ..."
Abstract

Cited by 19 (10 self)
 Add to MetaCart
With the global proliferation of wind power, accurate shortterm forecasts of wind resources at wind energy sites are becoming paramount. Regimeswitching spacetime (RST) models merge meteorological and statistical expertise to obtain accurate and calibrated, fully probabilistic forecasts of wind speed and wind power. The model formulation is parsimonious, yet takes account of all the salient features of wind speed: alternating atmospheric regimes, temporal and spatial correlation, diurnal and seasonal nonstationarity, conditional heteroscedasticity, and nonGaussianity. The RST method identifies forecast regimes at the wind energy site and fits a conditional predictive model for each regime. Geographically dispersed meteorological observations in the vicinity of the wind farm are used as offsite predictors. The RST technique was applied to 2hour ahead forecasts of hourly average wind speed at the Stateline wind farm in the US Pacific Northwest. In July 2003, for instance, the RST forecasts had rootmeansquare error (RMSE) 28.6 % less than the persistence forecasts. For each month in the test period, the RST forecasts had lower RMSE than forecasts using stateoftheart vector time series techniques. The RST method provides probabilistic forecasts in the form of
Sequential optimal design of neurophysiology experiments
, 2008
"... Adaptively optimizing experiments has the potential to significantly reduce the number of trials needed to build parametric statistical models of neural systems. However, application of adaptive methods to neurophysiology has been limited by severe computational challenges. Since most neurons are hi ..."
Abstract

Cited by 18 (6 self)
 Add to MetaCart
Adaptively optimizing experiments has the potential to significantly reduce the number of trials needed to build parametric statistical models of neural systems. However, application of adaptive methods to neurophysiology has been limited by severe computational challenges. Since most neurons are high dimensional systems, optimizing neurophysiology experiments requires computing highdimensional integrations and optimizations in real time. Here we present a fast algorithm for choosing the most informative stimulus by maximizing the mutual information between the data and the unknown parameters of a generalized linear model (GLM) which we want to fit to the neuronâ€™s activity. We rely on important logconcavity and asymptotic normality properties of the posterior to facilitate the required computations. Our algorithm requires only lowrank matrix manipulations and a 2dimensional search to choose the optimal stimulus. The average running time of these operations scales quadratically with the dimensionality of the GLM, making realtime adaptive experimental design feasible even for highdimensional stimulus and parameter spaces. For example, we
The Equivalence of Constrained and Weighted Designs in Multiple Objective Design Problems
 Journal of the American Statistical Association
, 1996
"... Several competing objectives may be relevant in the design of an experiment. The competing objectives may not be easy to characterize in a single optimality criterion. One approach to these design problems has been to weight each criterion and find the design that optimizes the weighted average of t ..."
Abstract

Cited by 16 (3 self)
 Add to MetaCart
Several competing objectives may be relevant in the design of an experiment. The competing objectives may not be easy to characterize in a single optimality criterion. One approach to these design problems has been to weight each criterion and find the design that optimizes the weighted average of the criteria. An alternative approach has been to optimize one criterion subject to constraints on the other criteria. An equivalence theorem is presented for the Bayesian constrained design problem. Equivalence theorems are essential in verifying optimality of proposed designs, especially when, as in most nonlinear design problems, numerical optimization is required. This theorem is used to show that the results of Cook and Wong on the equivalence of the weighted and constrained problems also apply much more generally. The results are applied to Bayesian nonlinear design problems with several objectives. KEY WORDS: Bayesian design, regression, nonlinear design 1. INTRODUCTION An experimen...
Relevance of Communicative acts
"... Why do we speak? Because we want to influence each other's behavior. The relevance of a speech act can measure its usefulness. In this paper I argue that (i) the relevance of a speech act depends on the `language game' one is involved in; (ii) notions of relevance can be defined using decision, info ..."
Abstract

Cited by 10 (1 self)
 Add to MetaCart
Why do we speak? Because we want to influence each other's behavior. The relevance of a speech act can measure its usefulness. In this paper I argue that (i) the relevance of a speech act depends on the `language game' one is involved in; (ii) notions of relevance can be defined using decision, information and game theory, and can be used for linguistic applications; and (iii) the strategic considerations of participants in a conversation deserve our attention, especially when we consider mixedmotive games of imperfect information, for instance, to establish the common ground.
Penalized loss functions for Bayesian model comparison
"... The deviance information criterion (DIC) is widely used for Bayesian model comparison, despite the lack of a clear theoretical foundation. DIC is shown to be an approximation to a penalized loss function based on the deviance, with a penalty derived from a crossvalidation argument. This approximati ..."
Abstract

Cited by 10 (0 self)
 Add to MetaCart
The deviance information criterion (DIC) is widely used for Bayesian model comparison, despite the lack of a clear theoretical foundation. DIC is shown to be an approximation to a penalized loss function based on the deviance, with a penalty derived from a crossvalidation argument. This approximation is valid only when the effective number of parameters in the model is much smaller than the number of independent observations. In disease mapping, a typical application of DIC, this assumption does not hold and DIC underpenalizes more complex models. Another deviancebased loss function, derived from the same decisiontheoretic framework, is applied to mixture models, which have previously been considered an unsuitable application for DIC.
The geometry of proper scoring rules
, 2007
"... A decision problem is defined in terms of an outcome space, an action space and a loss function. Starting from these simple ingredients, we ..."
Abstract

Cited by 9 (0 self)
 Add to MetaCart
A decision problem is defined in terms of an outcome space, an action space and a loss function. Starting from these simple ingredients, we
Applications of Lindley Information Measure to the Design of Clinical Experiments
 Aspects of Uncertainty
, 1994
"... this paper we consider applications of Lindley information measure to the design of clinical experiments. We review the decision theoretic foundations underlying the use of Lindley information, and discuss its role in constructing utility functions suitable for clinical applications. We derive and i ..."
Abstract

Cited by 8 (3 self)
 Add to MetaCart
this paper we consider applications of Lindley information measure to the design of clinical experiments. We review the decision theoretic foundations underlying the use of Lindley information, and discuss its role in constructing utility functions suitable for clinical applications. We derive and interpret general firstorder conditions for the optimality of a design. We discuss examples: choosing the optimal fixed sample size of a clinical trial, and choosing the optimal followup time for patients in a survival analysis. We give special attention to the design of multicenter clinical trials. Research of D. A. Berry supported in part by the US Public Health Service under grant HS 0647501. Research of Giovanni Parmigiani and ISDS computing environment supported in part by NSF under grant DMS9305699. We are thankful to Chengchang Li, Peter Muller, Saurabh Mukhopadhyay and Dalene Stangl for helpful discussions. 1. INTRODUCTION From the point of view of decision making, information is anything that enables us to make a better decision, that is a decision with a higher expected utility. For example, an experiment that, irrespective of the outcome, will lead to the same decision that we would make prior to observing it, has no information content. Conversely, experiments able to lead to different decision are potentially of benefit. The expected change in utility can actually be used as a quantitative measure of the worth of an experiment in any given situation. This idea is about as old as Bayesian statistics (see Ramsey, 1990) and is discussed by Raiffa and Schlaifer (1961) and DeGroot (1984). The well known measure of information proposed by Lindley (1956) is the object of investigation in this paper. It can be seen as a very important special case of this general ap...