In this work, we discuss practical methods for the assessment, comparison, and selection of complex hierarchical Bayesian models. A natural way to assess the goodness of the model is to estimate its future predictive capability by estimating expected utilities. Instead of just making a point estimate, it is important to obtain the distribution of the expected utility estimate, as it describes the uncertainty in the estimate. The distributions of the expected utility estimates can also be used to compare models, for example, by computing the probability of one model having a better expected utility than some other model. We propose an approach using crossvalidation predictive densities to obtain expected utility estimates and Bayesian bootstrap to obtain samples from their distributions. We also discuss the probabilistic assumptions made and properties of two practical cross-validation methods, importance sampling and k-fold cross-validation. As illustrative examples, we use MLP neural networks and Gaussian Processes (GP) with Markov chain Monte Carlo sampling in one toy problem and two challenging real-world problems.

When performing regression or classification, we are interested in the conditional probability distribution for an outcome or class variable Y given a set of explanatory or input variables X. We consider Bayesian models for this task. In particular, we examine a special class of models, which we call Bayesian regression/classification (BRC) models, that can be factored into independent conditional (yjx) and input (x) models. These models are convenient, because the conditional model (the portion of the full model that we care about) can be analyzed by itself. We examine the practice of transforming arbitrary Bayesian models to BRC models, and argue that this practice is often inappropriate because it ignores prior knowledge that may be important for learning. In addition, we examine Bayesian methods for learning models from data. We discuss two criteria for Bayesian model selection that are appropriate for repression/classification: one described by Spiegelhalter et al. (1993), and an...

this paper, we assume that there are a finite number of possible true models. For each possible model m, we define the random (vector) variable \Theta m whose values correspond to the possible values of the parameters for m. We encode our uncertainty about \Theta m using the probability distribution p(\Theta m jm). In this paper, we assume that p(\Theta m jm) is a probability density function. Given random sample D, we compute the posterior distributions for M and each \Theta m

With the global proliferation of wind power, accurate short-term forecasts of wind resources at wind energy sites are becoming paramount. Regime-switching space-time (RST) models merge meteorological and statistical expertise to obtain accurate and calibrated, fully probabilistic forecasts of wind speed and wind power. The model formulation is parsimonious, yet takes account of all the salient features of wind speed: alternating atmospheric regimes, temporal and spatial correlation, diurnal and seasonal non-stationarity, conditional heteroscedasticity, and non-Gaussianity. The RST method identifies forecast regimes at the wind energy site and fits a conditional predictive model for each regime. Geographically dispersed meteorological observations in the vicinity of the wind farm are used as off-site predictors. The RST technique was applied to 2-hour ahead forecasts of hourly average wind speed at the Stateline wind farm in the US Pacific Northwest. In July 2003, for instance, the RST forecasts had root-mean-square error (RMSE) 28.6 % less than the persistence forecasts. For each month in the test period, the RST forecasts had lower RMSE than forecasts using state-of-the-art vector time series techniques. The RST method provides probabilistic forecasts in the form of

Several competing objectives may be relevant in the design of an experiment. The competing objectives may not be easy to characterize in a single optimality criterion. One approach to these design problems has been to weight each criterion and find the design that optimizes the weighted average of the criteria. An alternative approach has been to optimize one criterion subject to constraints on the other criteria. An equivalence theorem is presented for the Bayesian constrained design problem. Equivalence theorems are essential in verifying optimality of proposed designs, especially when, as in most nonlinear design problems, numerical optimization is required. This theorem is used to show that the results of Cook and Wong on the equivalence of the weighted and constrained problems also apply much more generally. The results are applied to Bayesian nonlinear design problems with several objectives. KEY WORDS: Bayesian design, regression, nonlinear design 1. INTRODUCTION An experimen...

this paper we consider applications of Lindley information measure to the design of clinical experiments. We review the decision theoretic foundations underlying the use of Lindley information, and discuss its role in constructing utility functions suitable for clinical applications. We derive and interpret general first-order conditions for the optimality of a design. We discuss examples: choosing the optimal fixed sample size of a clinical trial, and choosing the optimal follow-up time for patients in a survival analysis. We give special attention to the design of multicenter clinical trials. Research of D. A. Berry supported in part by the US Public Health Service under grant HS 06475-01. Research of Giovanni Parmigiani and ISDS computing environment supported in part by NSF under grant DMS-9305699. We are thankful to Chengchang Li, Peter Muller, Saurabh Mukhopadhyay and Dalene Stangl for helpful discussions. 1. INTRODUCTION From the point of view of decision making, information is anything that enables us to make a better decision, that is a decision with a higher expected utility. For example, an experiment that, irrespective of the outcome, will lead to the same decision that we would make prior to observing it, has no information content. Conversely, experiments able to lead to different decision are potentially of benefit. The expected change in utility can actually be used as a quantitative measure of the worth of an experiment in any given situation. This idea is about as old as Bayesian statistics (see Ramsey, 1990) and is discussed by Raiffa and Schlaifer (1961) and DeGroot (1984). The well known measure of information proposed by Lindley (1956) is the object of investigation in this paper. It can be seen as a very important special case of this general ap...

Why do we speak? Because we want to influence each other's behavior. The relevance of a speech act can measure its usefulness. In this paper I argue that (i) the relevance of a speech act depends on the `language game' one is involved in; (ii) notions of relevance can be defined using decision, information and game theory, and can be used for linguistic applications; and (iii) the strategic considerations of participants in a conversation deserve our attention, especially when we consider mixed-motive games of imperfect information, for instance, to establish the common ground.

Abstract not found

Bayesian optimal designs for nonlinear regression models are of some interest and importance in the statistical literature. Numerical methods for their construction are well-established, but very few analytical studies have been reported. In this paper, we consider an exponential growth model used extensively in the modelling of simple organisms, and examine the explicit form of the Bayesian D--optimal designs. In particular, we show that D ` --optimal designs for this model are balanced two--point designs for all values of the parameters. We further derive explicit expressions for Bayesian D--optimal designs which are based on exactly two points of support, and provide necessary and sufficient conditions for such designs to exist. We illustrate our results by means of two examples. Key Words: Nonlinear regression; exponential growth model; D--optimality; locally optimal designs; Bayesian design. AMS Subject Classification: Primary 62K05; Secondary 62F15 1 Introduction Many authors ...

We consider the input variable selection in complex Bayesian hierarchical models. Our goal is to find a model with the smallest number of input variables having statistically or practically at least the same expected utility as the full model with all the available inputs. A good estimate for the expected utility can be computed using cross-validation predictive densities. In the case of input selection and a large number of input combinations, the computation of the cross-validation predictive densities for each model easily becomes computationally prohibitive. We propose to use the posterior probabilities obtained via variable dimension MCMC methods to find out potentially useful input combinations, for which the final model choice and assessment is done using the expected utilities.