Scoring rules assess the quality of probabilistic forecasts, by assigning a numerical score based on the predictive distribution and on the event or value that materializes. A scoring rule is proper if the forecaster maximizes the expected score for an observation drawn from the distribution F if he or she issues the probabilistic forecast F, rather than G ̸ = F. It is strictly proper if the maximum is unique. In prediction problems, proper scoring rules encourage the forecaster to make careful assessments and to be honest. In estimation problems, strictly proper scoring rules provide attractive loss and utility functions that can be tailored to the problem at hand. This article reviews and develops the theory of proper scoring rules on general probability spaces, and proposes and discusses examples thereof. Proper scoring rules derive from convex functions and relate to information measures, entropy functions, and Bregman divergences. In the case of categorical variables, we prove a rigorous version of the Savage representation. Examples of scoring rules for probabilistic forecasts in the form of predictive densities include the logarithmic, spherical, pseudospherical, and quadratic scores. The continuous ranked probability score applies to probabilistic forecasts that take the form of predictive cumulative distribution functions. It generalizes the absolute error and forms a special case of a new and very general type of score, the energy score. Like many other scoring rules, the energy score admits a kernel representation in terms of negative definite functions, with links to inequalities of Hoeffding type, in both univariate and multivariate settings. Proper scoring rules for quantile and interval forecasts are also discussed. We relate proper scoring rules to Bayes factors and to cross-validation, and propose a novel form of cross-validation known as random-fold cross-validation. A case study on probabilistic weather forecasts in the North American Pacific Northwest illustrates the importance of propriety. We note optimum score approaches to point and quantile

Summary. Probabilistic forecasts of continuous variables take the form of predictive densities or predictive cumulative distribution functions. We propose a diagnostic approach to the evaluation of predictive performance that is based on the paradigm of maximizing the sharpness of the predictive distributions subject to calibration. Calibration refers to the statistical consistency between the distributional forecasts and the observations and is a joint property of the predictions and the events that materialize. Sharpness refers to the concentration of the predictive distributions and is a property of the forecasts only. A simple theoretical framework allows us to distinguish between probabilistic calibration, exceedance calibration and marginal calibration. We propose and study tools for checking calibration and sharpness, among them the probability integral transform histogram, marginal calibration plots, the sharpness diagram and proper scoring rules. The diagnostic approach is illustrated by an assessment and ranking of probabilistic forecasts of wind speed at the Stateline wind energy centre in the US Pacific Northwest. In combination with cross-validation or in the time series context, our proposal provides very general, nonparametric alternatives to the use of information criteria for model diagnostics and model selection.

Projections of future climate change caused by increasing greenhouse gases depend critically on numerical climate models coupling the ocean and atmosphere (GCMs). However, different models differ substantially in their projections, which raises the question of how the different models can best be combined into a probability distribution of future climate change. For this analysis, we have collected both current and future projected mean temperatures produced by nine climate models for 22 regions of the earth. We also have estimates of current mean temperatures from actual observations, together with standard errors, that can be used to calibrate the climate models. We propose a Bayesian analysis that allows us to combine the different climate models into a posterior distribution of future temperature increase, for each of the 22 regions, while allowing for the different climate models to have different variances. Two versions of the analysis are proposed, a univariate analysis in which each region is analyzed separately, and a multivariate analysis in which the 22 regions are combined into an overall statistical model. A cross-validation approach is proposed to confirm the reasonableness of our Bayesian predictive distributions. The results of this analysis allow for a quantification of the uncertainty of climate model projections as a Bayesian posterior distribution, substantially extending previous approaches to uncertainty in climate models.

the Joint Ensemble Forecasting System (JEFS) under subcontract S06-47225 from the University

Wind direction is an angular variable, as opposed to weather quantities such as temperature, quantitative precipitation or wind speed, which are linear variables. Consequently, traditional model output statistics and ensemble post-processing methods become ineffective, or do not apply at all. We propose an effective bias correction technique for wind direction forecasts from numerical weather prediction models, which is based on a state-of-the-art circular-circular regression approach. To calibrate forecast ensembles, a Bayesian model averaging scheme for directional variables is introduced, where the component distributions are von Mises densities centered at the individually bias-corrected ensemble member forecasts. We apply these techniques to 48-hour forecasts of surface wind direction over the Pacific Northwest, using the University of Washington Mesoscale Ensemble, where they yield consistent improvements in forecast performance. 1

As wind energy penetration continues to grow, there is a critical need for probabilistic forecasts of wind resources. In addition, there are many other societally relevant uses for forecasts of wind speed, ranging from aviation to ship routing and recreational boating. Over the past two decades, ensembles of numerical weather prediction (NWP) models have been developed, in which multiple estimates of the current state of the atmosphere are used to generate a collection of deterministic forecasts. However, even state-of-the-art ensemble systems are uncalibrated and biased. Here we propose a novel way of statistically post-processing NWP ensembles for wind speed using heteroskedastic censored (Tobit) regression, where location and spread derive from the ensemble forecast. The resulting ensemble model output statistics (EMOS) method is applied to 48-hour ahead forecasts of maximum wind speed over the North American Pacific Northwest in 2003 using the University of Washington Mesoscale Ensemble. The statistically post-processed EMOS density forecasts turn out to be calibrated and sharp, and result in substantial improvement over the unprocessed NWP ensemble or climatological reference forecasts.

Scoring rules assess the quality of probabilistic forecasts, by assigning a numerical score based on the predictive distribution and on the event or value that materializes. A scoring rule is proper if it encourages truthful reporting. It is local of order λ if the score depends on the predictive density only through its value and its derivatives of order up to λ at the observation. Previously, only a single local proper scoring rule had been known, namely the logarithmic score, which is local of order λ = 0. Here we introduce the Fisher score, which is a local proper scoring rule of order λ = 2. It relates to the Fisher information in the same way that the logarithmic score relates to the Kullback-Leibler information. The convex cone generated by the logarithmic score and the Fisher score exhausts the class of the local proper scoring rules of order λ ≤ 2, up to equivalence and regularity conditions. In a data example, we use local and non-local proper scoring rules to assess statistically postprocessed ensemble weather forecasts. Finally, we develop a multivariate version of the Fisher score. 1

We present neural network surrogates that provide extremely fast and accurate emulation of a large-scale circulation model for the coupled Columbia River, its estuary and near ocean regions. The circulation model has O(107) degrees of freedom, is highly nonlinear and is driven by ocean, atmospheric and river influences at its boundaries. The surrogates provide accurate emulation of the full circulation code and run over 1000 times faster. Such fast dynamic surrogates will enable significant advances in ensemble forecasts in oceanography and weather.

Optimal probabilistic forecasts of integer-valued random variables are derived. The optimality is achieved by estimating the forecast distribution nonparametrically over a given broad model class and proving asymptotic efficiency in that setting. The ideas are demonstrated within the context of the integer autoregressive class of models, which is a suitable class for any count data that can be interpreted as a queue, stock, birth and death process or branching process. The theoretical proofs of asymptotic optimality are supplemented by simulation results which demonstrate the overall superiority of the nonparametric method relative to a misspecified parametric maximum likelihood estimator, in large but finite samples. The method is applied to counts of wage claim benefits, stock market iceberg orders and civilian deaths in Iraq, with bootstrap methods used to quantify sampling variation in the estimated forecast distributions.

Abstract. This note proves a weak type of the sharpness principle as conjectured by Gneiting, Balabdaoui, and Raftery [9] in connection with probabilistic forecasting subject to calibration constraints. A strong version of such a principle still awaits a proper formulation. 1.