This paper reviews the literature on Bayesian experimental design, both for linear and nonlinear models. A unified view of the topic is presented by putting experimental design in a decision theoretic framework. This framework justifies many optimality criteria, and opens new possibilities. Various design criteria become part of a single, coherent approach.

Introduction Deterministic computer simulations of physical phenomena are becoming widely used in science and engineering. Computers are used to describe the flow of air over an airplane wing, combustion of gasses in a flame, behavior of a metal structure under stress, safety of a nuclear reactor, and so on. Some of the most widely used computer models, and the ones that lead us to work in this area, arise in the design of the semiconductors used in the computers themselves. A process simulator starts with a data structure representing an unprocessed piece of silicon and simulates the steps such as oxidation, etching and ion injection that produce a semiconductor device such as a transistor. A device simulator takes a description of such a device and simulates the flow of current through it under varying conditions to determine properties of the device such as its switching speed and the critical voltage at which it switches. A circuit simulator takes a list of devices and the

this paper a Bayesian approach to the calibration of computer models. We represent the unknown inputs as a parameter vector `. Using the observed data we derive the posterior distribution of `, which in particular quantifies the `residual uncertainty' about

This paper builds on work by Haylock and O'Hagan which developed a Bayesian approach to uncertainty analysis. The generic problem is to make posterior inference about the output of a complex computer code, and the specific problem of uncertainty analysis is to make inference when the "true" values of the input parameters are unknown. Given the distribution of the input parameters (which is often a subjective distribution derived from expert opinion), we wish to make inference about the implied distribution of the output. The computer code is sufficiently complex that the time to compute the output for any input configuration is substantial. The Bayesian approach was shown to improve dramatically on the classical approach, which is based on drawing a sample of values of the input parameters and thereby obtaining a sample from the output distribution. We review the basic Bayesian approach to the generic problem of inference for complex computer codes, and present some recent advances---inference about the distribution of quantile functions of the uncertainty distribution, calibration of models, and the use of runs of the computer code at different levels of complexity to make efficient use of the quicker, cruder, versions of the code. The emphasis is on practical applications. Keywords: COMPUTATIONAL EXPERIMENT; SIMULATION; GAUSSIAN PROCESS; SENSITIVITY ANALYSIS; UNCERTAINTY DISTRIBUTION; CALIBRATION; MULTI-LEVEL CODES; MODEL INADEQUACY. 1. INTRODUCTION 1.1. Complex computer codes

The paper summarizes some important results at the intersection of the fields of Bayesian statistics and stochastic simulation. Two statistical analysis issues for stochastic simulation are discussed in further detail from a Bayesian perspective. First, a review of recent work in input distribution selection is presented. Then, a new Bayesian formulation for the problem of output analysis for a single system is presented. A key feature is analyzing simulation output as a random variable whose parameters are an unknown function of the simulation's inputs. The distribution of those parameters is inferred from simulation output via Bayesian response-surface methods. A brief summary of Bayesian inference and decision making is included for reference.

Computer-based simulation and analysis is used extensively in engineering for a variety of tasks. Despite the steady and continuing growth of computing power and speed, the computational cost of complex high-fidelity engineering analyses and simulations limit their use in important areas like design optimization and reliability analysis. Statistical approximation techniques such as design of experiments and response surface methodology are becoming widely used in engineering to minimize the computational expense of running such computer analyses and circumvent many of these limitations. In this paper, we compare and contrast five experimental design types and four approximation model types in terms of their capability to generate accurate approximations for two engineering applications with typical engineering behaviors and a wide range of nonlinearity. The first example involves the analysis of a two-member frame that has three input variables and three responses of interest. The second example simulates the roll-over potential of a semi-tractor-trailer for different combinations of input variables and braking and steering levels. Detailed error analysis reveals that uniform designs provide good sampling for generating accurate approximations using different sample sizes while kriging models provide accurate approximations that are robust for use with a variety of experimental designs and sample sizes.

In this paper, we compare and contrast the use of second-order response surface models and kriging models for approximating non-random, deterministic computer analyses. After reviewing the response surface method for constructing polynomial approximations, kriging is presented as an alternative approximation method for the design and analysis of computer experiments. Both methods are applied to the multidisciplinary design of an aerospike nozzle which consists of a computational fluid dynamics model and a finite-element model. Error analysis of the response surface and kriging models is performed along with a graphical comparison of the approximations, and four optimization problems are formulated and solved using both sets of approximation models. The second-order response surface models and kriging models---using a constant underlying global model and a Gaussian correlation function---yield comparable results. NOMENCLATURE b - constant underlying global portion of kriging model b ...

In many scienti � c and engineering domains, it is common to analyze and simulate complex physical systems using mathematical models. Although computing resources continue to increase in power and speed, computer simulation and analysis codes continue to grow in complexity and remain computationally expensive, limiting their use in design and optimization. Consequently, many researchers have developed different metamodeling strategies to create inexpensive approximations of computationally expensive computer simulations. These approximations introduce a new element of uncertainty during design optimization, and there is a need to develop ef � cient methods to assess metamodel validity. We investigate computationally inexpensive assessment methods for metamodel validation based on leave-k-out cross validation and develop guidelines for selecting k for different types of metamodels. Based on the results from two sets of test problems, k = 1 is recommended for leave-k-out cross validation of loworder polynomial and radial basis function metamodels, whereas k = 0:1N or N is recommended for kriging metamodels, where N is the number of sample points used to construct the metamodel. Nomenclature N = number of sample points x = design (input) variable y = actual output (response) value Oyi = predicted output (response) value from metamodel I.

real-world phenomena. They are typically used to predict the corresponding real-world phenomenon, as in the following examples. Modern weather forecasting is done using enormously complex models of the atmosphere (and its interactions with land and sea). The primary intention is to predict future weather, given information about current conditions. Manufacturers of motor car engines build models to predict their behaviour. They are used to explore possible variations in engine design, and thereby to avoid the time and expense of actually building many unsuccessful variants in the search for an improved design. Water engineers build network ‡ow models of sewer systems, in order to predict where problems of surcharging and ‡ooding will arise under rainstorm conditions. They are then used to explore changes to the network to solve those problems. Models of atmospheric dispersion are used to predict the spread and deposition

We consider prediction and uncertainty analysis for systems which are approximated using complex mathematical codes. Such models, implemented as computer codes, are often generic in the sense that by suitable choice of some of the model's input parameters the code can be used to predict behaviour of the system in a variety of specific applications. However, in any specific application the values of necessary parameters may be unknown. In this case, physical observations of the system in the specific context are used to learn about the unknown parameters. The process of fitting the model to the observed data by adjusting the parameters is known as calibration. Calibration is typically effected by ad hoc fitting, and after calibration the model is used, with the fitted input values, to predict future behaviour of the system. We present a Bayesian calibration technique which improves on this traditional approach in two respects. First, the predictions allow for remaining uncertainty over ...