A theory for estimating the probability distribution of the state of a model given a set of observations exists. This nonlinear

Ensembles used for probabilistic weather forecasting often exhibit a spread-error correlation, but they tend to be underdispersive. This paper proposes a statistical method for postprocessing ensembles based on Bayesian model averaging (BMA), which is a standard method for combining predictive distributions from different sources. The BMA predictive probability density function (PDF) of any quantity of interest is a weighted average of PDFs centered on the individual bias-corrected forecasts, where the weights are equal to posterior probabilities of the models generating the forecasts and reflect the models ’ relative contributions to predictive skill over the training period. The BMA weights can be used to assess the usefulness of ensemble members, and this can be used as a basis for selecting ensemble members; this can be useful given the cost of running large ensembles. The BMA PDF can be represented as an unweighted ensemble of any desired size, by simulating from the BMA predictive distribution. The BMA predictive variance can be decomposed into two components, one corresponding to the between-forecast variability, and the second to the within-forecast variability. Predictive PDFs or intervals based solely on the ensemble spread incorporate the first component but not the second. Thus BMA provides a theoretical explanation of the tendency of ensembles to exhibit a spread-error correlation but yet

Rank histograms are a tool for evaluating ensemble forecasts. They are useful for determining the reliability of ensemble forecasts and for diagnosing errors in its mean and spread. Rank histograms are generated by repeatedly tallying the rank of the verification (usually, an observation) relative to values from an ensemble sorted from lowest to highest. However, an uncritical use of the rank histogram can lead to misinterpretations of the qualities of that ensemble. For example, a flat rank histogram, ususally taken as a sign of reliability, can still be generated from unreliable ensembles. Similarly, a U-shaped rank histogram, commonly understood as indicating a lack of variability in the ensemble, can also be a sign of conditional bias. It is also shown that flat rank histograms can be generated for some model variables if the variance of the ensemble is correctly specified, yet if covariances between model grid points are improperly specified, rank histograms for combinations of mo...

Summary. Probabilistic forecasts of continuous variables take the form of predictive densities or predictive cumulative distribution functions. We propose a diagnostic approach to the evaluation of predictive performance that is based on the paradigm of maximizing the sharpness of the predictive distributions subject to calibration. Calibration refers to the statistical consistency between the distributional forecasts and the observations and is a joint property of the predictions and the events that materialize. Sharpness refers to the concentration of the predictive distributions and is a property of the forecasts only. A simple theoretical framework allows us to distinguish between probabilistic calibration, exceedance calibration and marginal calibration. We propose and study tools for checking calibration and sharpness, among them the probability integral transform histogram, marginal calibration plots, the sharpness diagram and proper scoring rules. The diagnostic approach is illustrated by an assessment and ranking of probabilistic forecasts of wind speed at the Stateline wind energy centre in the US Pacific Northwest. In combination with cross-validation or in the time series context, our proposal provides very general, nonparametric alternatives to the use of information criteria for model diagnostics and model selection.

Chaos places no a priori restrictions on predictability: any uncertainty in the initial condition can be evolved and then quanti ed as a function of forecast time. If a speci ed accuracy at a given future time is desired, a perfect model can specify the initial accuracy required to obtain it, and accountable ensemble forecasts can be obtained for each unknown initial condition. Statistics which reect the global properties of in nitesimals, such as Lyapunov exponents which de ne \chaos", limit predictability only in the simplest mathematical examples. Model error, on the other hand, makes forecasting a dubious endeavor. Forecasting with uncertain initial conditions in the perfect model scenario is contrasted with the case where a perfect model is unavailable, perhaps nonexistent. Applications to both low (2 to 400) dimensional models and high (10 7 ) dimensional models are discussed. For real physical systems no perfect model exists; the limitations of nearperfect models are consider...

Abstract not found

and useful comments, and for providing data. They are also grateful to Patrick Tewson for implementing the UW Ensemble BMA website. This research was supported by the DoD Multidisciplinary University Research Initiative (MURI) program administered by the Office of Naval Research under Grant N00014-01-10745. Bayesian model averaging (BMA) is a statistical way of postprocessing forecast ensembles to create predictive probability density functions (PDFs) for weather quantities. It represents the predictive PDF as a weighted average of PDFs centered on the individual bias-corrected forecasts, where the weights are posterior probabilities of the models generating the forecasts and reflect the forecasts ’ relative contributions to predictive skill over a training period. It was developed initially for quantities whose PDFs can be approximated by normal distributions, such as temperature and sea-level pressure. BMA does not apply in its original form to precipitation, because the predictive PDF of precipitation is nonnormal in two major ways: it has a positive probability of being equal to zero, and it is skewed. Here we extend BMA to probabilistic quantitative precipitation forecasting. The predictive PDF corresponding to

Ensemble prediction systems typically show positive spread-error correlation, but they are subject to forecast bias and underdispersion, and therefore uncalibrated. This work proposes the use of ensemble model output statistics (EMOS), an easy to imple-ment post-processing technique that addresses both forecast bias and underdispersion and takes account of the spread-skill relationship. The technique is based on multiple lin-ear regression and akin to the superensemble approach that has traditionally been used for deterministic-style forecasts. The EMOS technique yields probabilistic forecasts that take the form of Gaussian predictive probability density functions (PDFs) for continuous weather variables, and can be applied to gridded model output. The EMOS predictive mean is an optimal, bias-corrected weighted average of the ensemble member forecasts, with coefficients that are constrained to be nonnegative and associated with the member model skill. The EMOS predictive mean provides a highly accurate deterministic-style forecast. The EMOS predictive variance is a linear function of the ensemble spread. For fitting the EMOS coefficients, the method of minimum CRPS estimation is introduced.

We develop a method for assessing uncertainty about quantities of interest using urban simulation models. The method is called Bayesian melding, and extends a previous method developed for macrolevel deterministic simulation models to agent-based stochastic models. It encodes all the available information about model inputs and outputs in terms of prior probability distributions and likelihoods, and uses Bayes’s theorem to obtain the resulting posterior distribution of any quantity of interest that is a function of model inputs and/or outputs. It is Monte Carlo based, and quite easy to implement. We applied it to the projection of future household numbers by traffic activity zone in Eugene-Springfield, Oregon, using the UrbanSim model developed at the University of Washington. We compared it with a simpler method that uses repeated runs of the model with fixed estimated inputs. We found that the simple repeated runs method gave distributions of quantities of interest that were too narrow, while Bayesian melding gave well calibrated uncertainty statements.

A statistical model and extended ensemble integrations of two atmospheric general circulation models (GCMs) are used to simulate the extratropical atmospheric response to forcing by observed SSTs for the years 1980 through 1988. The simulations are compared to observations using the anomaly correlation and root-mean-square error of the 700-hPa height field over a region encompassing the extratropical North Pacific Ocean and most of North America. On average, the statistical model is found to produce considerably better simulations than either numerical model, even when simple statistical corrections are used to remove systematic errors from the numerical model simulations. In the mean, the simulation skill is low, but there are some individual seasons for which all three models produce simulations with good skill. An approximate upper bound to the simulation skill that could be expected from a GCM ensemble, if the model’s response to SST forcing is assumed to be perfect, is computed. This perfect model predictability allows one to make some rough extrapolations about the skill that could be expected if one could greatly improve the mean response of the GCMs without significantly impacting the variance of the ensemble. These perfect model predictability skills are better than the statistical model simulations during the summer, but for the winter, present-day statistical forecasts already have skill that is as high as the upper bound for the GCMs. Simultaneous improvements to the GCM mean response and reduction in the GCM ensemble variance would be required for these GCMs to do significantly better than the statistical