Results 1  10
of
22
Strictly Proper Scoring Rules, Prediction, and Estimation
, 2007
"... Scoring rules assess the quality of probabilistic forecasts, by assigning a numerical score based on the predictive distribution and on the event or value that materializes. A scoring rule is proper if the forecaster maximizes the expected score for an observation drawn from the distribution F if he ..."
Abstract

Cited by 373 (28 self)
 Add to MetaCart
Scoring rules assess the quality of probabilistic forecasts, by assigning a numerical score based on the predictive distribution and on the event or value that materializes. A scoring rule is proper if the forecaster maximizes the expected score for an observation drawn from the distribution F if he or she issues the probabilistic forecast F, rather than G ̸ = F. It is strictly proper if the maximum is unique. In prediction problems, proper scoring rules encourage the forecaster to make careful assessments and to be honest. In estimation problems, strictly proper scoring rules provide attractive loss and utility functions that can be tailored to the problem at hand. This article reviews and develops the theory of proper scoring rules on general probability spaces, and proposes and discusses examples thereof. Proper scoring rules derive from convex functions and relate to information measures, entropy functions, and Bregman divergences. In the case of categorical variables, we prove a rigorous version of the Savage representation. Examples of scoring rules for probabilistic forecasts in the form of predictive densities include the logarithmic, spherical, pseudospherical, and quadratic scores. The continuous ranked probability score applies to probabilistic forecasts that take the form of predictive cumulative distribution functions. It generalizes the absolute error and forms a special case of a new and very general type of score, the energy score. Like many other scoring rules, the energy score admits a kernel representation in terms of negative definite functions, with links to inequalities of Hoeffding type, in both univariate and multivariate settings. Proper scoring rules for quantile and interval forecasts are also discussed. We relate proper scoring rules to Bayes factors and to crossvalidation, and propose a novel form of crossvalidation known as randomfold crossvalidation. A case study on probabilistic weather forecasts in the North American Pacific Northwest illustrates the importance of propriety. We note optimum score approaches to point and quantile
Probabilistic forecasts, calibration and sharpness
 Journal of the Royal Statistical Society Series B
, 2007
"... Summary. Probabilistic forecasts of continuous variables take the form of predictive densities or predictive cumulative distribution functions. We propose a diagnostic approach to the evaluation of predictive performance that is based on the paradigm of maximizing the sharpness of the predictive dis ..."
Abstract

Cited by 116 (23 self)
 Add to MetaCart
Summary. Probabilistic forecasts of continuous variables take the form of predictive densities or predictive cumulative distribution functions. We propose a diagnostic approach to the evaluation of predictive performance that is based on the paradigm of maximizing the sharpness of the predictive distributions subject to calibration. Calibration refers to the statistical consistency between the distributional forecasts and the observations and is a joint property of the predictions and the events that materialize. Sharpness refers to the concentration of the predictive distributions and is a property of the forecasts only. A simple theoretical framework allows us to distinguish between probabilistic calibration, exceedance calibration and marginal calibration. We propose and study tools for checking calibration and sharpness, among them the probability integral transform histogram, marginal calibration plots, the sharpness diagram and proper scoring rules. The diagnostic approach is illustrated by an assessment and ranking of probabilistic forecasts of wind speed at the Stateline wind energy centre in the US Pacific Northwest. In combination with crossvalidation or in the time series context, our proposal provides very general, nonparametric alternatives to the use of information criteria for model diagnostics and model selection.
Calibrated Probabilistic Forecasting Using Ensemble Model Output Statistics and Minimum CRPS Estimation
 MONTHLY WEATHER REVIEW VOLUME
, 2005
"... Ensemble prediction systems typically show positive spreaderror correlation, but they are subject to forecast bias and dispersion errors, and are therefore uncalibrated. This work proposes the use of ensemble model output statistics (EMOS), an easytoimplement postprocessing technique that address ..."
Abstract

Cited by 81 (14 self)
 Add to MetaCart
Ensemble prediction systems typically show positive spreaderror correlation, but they are subject to forecast bias and dispersion errors, and are therefore uncalibrated. This work proposes the use of ensemble model output statistics (EMOS), an easytoimplement postprocessing technique that addresses both forecast bias and underdispersion and takes into account the spreadskill relationship. The technique is based on multiple linear regression and is akin to the superensemble approach that has traditionally been used for deterministicstyle forecasts. The EMOS technique yields probabilistic forecasts that take the form of Gaussian predictive probability density functions (PDFs) for continuous weather variables and can be applied to gridded model output. The EMOS predictive mean is a biascorrected weighted average of the ensemble member forecasts, with coefficients that can be interpreted in terms of the relative contributions of the member models to the ensemble, and provides a highly competitive deterministicstyle forecast. The EMOS predictive variance is a linear function of the ensemble variance. For fitting the EMOS coefficients, the method of minimum continuous ranked probability score (CRPS) estimation is introduced. This technique finds the coefficient values that optimize the CRPS for the training data. The EMOS technique was applied to 48h forecasts of sea level pressure and surface temperature over the North American Pacific Northwest in spring 2000, using the University of Washington mesoscale ensemble. When compared to the biascorrected ensemble, deterministicstyle EMOS forecasts of sea level pressure had rootmeansquare error 9 % less and mean absolute error 7 % less. The EMOS predictive PDFs were sharp, and much better calibrated than the raw ensemble or the biascorrected ensemble.
Default Priors and Predictive Performance in Bayesian Model Averaging, with Application to Growth Determinants
 Journal of Applied Econometrics
, 2011
"... Abstract Bayesian model averaging (BMA) has become widely accepted as a way of accounting for model uncertainty, notably in regression models for identifying the determinants of economic growth. To implement BMA the user must specify a prior distribution in two parts: a prior for the regression par ..."
Abstract

Cited by 30 (7 self)
 Add to MetaCart
(Show Context)
Abstract Bayesian model averaging (BMA) has become widely accepted as a way of accounting for model uncertainty, notably in regression models for identifying the determinants of economic growth. To implement BMA the user must specify a prior distribution in two parts: a prior for the regression parameters and a prior over the model space. Here we address the issue of which default prior to use for BMA in linear regression. We compare 12 candidate parameter priors: the Unit Information Prior (UIP) corresponding to the BIC or Schwarz approximation to the integrated likelihood, a proper datadependent prior, and 10 priors considered by Fernandez et al. (2001b). We also compare the uniform model prior to others that favor smaller models. We compare them on the basis of crossvalidated predictive performance on a wellknown growth dataset and on two simulated examples from the literature. We found that the UIP with uniform model prior generally outperformed the other priors considered. It also identified the largest set of growth determinants. JEL Classification: O51, O52, O53.
Do highfrequency measures of volatility improve forecasts of return distributions?
 JOURNAL OF ECONOMETRICS
"... ..."
Evaluating density forecast models
, 2006
"... Abstract. Density forecasting in regression is gaining popularity as real world applications demand an estimate of the level of uncertainty in predictions. In this paper we describe the two goals of density forecasting 1 sharpness and calibration. We review the evaluation methods available to a dens ..."
Abstract

Cited by 4 (0 self)
 Add to MetaCart
(Show Context)
Abstract. Density forecasting in regression is gaining popularity as real world applications demand an estimate of the level of uncertainty in predictions. In this paper we describe the two goals of density forecasting 1 sharpness and calibration. We review the evaluation methods available to a density forecaster to assess each of these goals and we introduce a new evaluation method that allows modelers to compare and evaluate their models across both of these goals simultaneously and identify the optimal model. 1
2010), “Modeling Realized Covariances and Returns
"... This paper proposes new dynamic component models of returns and realized covariance (RCOV) matrices based on timevarying Wishart distributions. Bayesian estimation and model comparison is conducted with a range of multivariate GARCH models and existing RCOV models from the literature. The main meth ..."
Abstract

Cited by 2 (0 self)
 Add to MetaCart
(Show Context)
This paper proposes new dynamic component models of returns and realized covariance (RCOV) matrices based on timevarying Wishart distributions. Bayesian estimation and model comparison is conducted with a range of multivariate GARCH models and existing RCOV models from the literature. The main method of model comparison consists of a termstructure of density forecasts of returns for multiple forecast horizons. The new joint returnRCOV models provide superior density forecasts for returns from forecast horizons of 1 day to 3 months ahead as well as improved point forecasts for realized covariances. Global minimum variance portfolio selection is improved for forecast horizons up to 3 weeks out. key words: Wishart distribution, predictive likelihoods, density forecasts, realized covariance targeting, MCMC.
Making Density Forecasting Models Statistically Consistent
"... Abstract. We propose a new approach to density forecast optimisation and apply it to ValueatRisk estimation. All existing density forecasting models try to optimise the distribution of the returns based solely on the predicted density at the observation. In this paper we argue that probabilistic p ..."
Abstract

Cited by 1 (1 self)
 Add to MetaCart
Abstract. We propose a new approach to density forecast optimisation and apply it to ValueatRisk estimation. All existing density forecasting models try to optimise the distribution of the returns based solely on the predicted density at the observation. In this paper we argue that probabilistic predictions should be optimised on more than just this accuracy score and suggest that the statistical consistency of the probability estimates should also be optimised during training. Statistical consistency refers to the property that if a predicted density function suggests P percent probability of occurrence, the event truly ought to have probability P of occurring. We describe a quality score that can rank probability density forecasts in terms of statistical consistency based on the probability integral transform (Diebold et al., 1998b). We then describe a framework that can optimise any density forecasting model in terms of any set of objective functions. The framework uses a multiobjective evolutionary algorithm to determine a set of tradeoff solutions known as the Pareto front of optimal solutions. Using this framework we develop an algorithm for optimising density forecasting models and implement this algorithm for GARCH (Bollerslev, 1986) and GJR models (Glosten et al., 1993). We call these new models ParetoGARCH and ParetoGJR. To determine whether this approach of multiobjective optimisation of density forecasting models produces better results over the standard GARCHand GJR optimisation techniques we compare the models produced empirically on a ValueatRisk application. Our evaluation shows that our Pareto models produce superior results outofsample.
The benefits of using a complete probability distribution when decision making: an example in anticoagulant drug therapy. Medical Decision Making (forthcoming
"... Abstract. In this paper we aim to show how probabilistic prediction of a continuous variable could be more beneficial to a medical practitioner than classification or numeric/point prediction of the same variable in many scenarios. We introduce a probability density forecasting model that produces a ..."
Abstract

Cited by 1 (0 self)
 Add to MetaCart
Abstract. In this paper we aim to show how probabilistic prediction of a continuous variable could be more beneficial to a medical practitioner than classification or numeric/point prediction of the same variable in many scenarios. We introduce a probability density forecasting model that produces accurate estimates and achieves statistically consistent predicted distributions. An empirical evaluation of this approach on the problem of warfarin dosage prediction is described and a comparison of results obtained from our probabilistic models with a number of classification techniques on this problem is also shown. 1
Oil prices  Brownian motion or mean reversion? A study using a one year ahead density forecast criterion
 Energy Economics
, 2010
"... Abstract. For oil related investment appraisal, an accurate description of the evolving uncertainty in the oil price is essential. For example, when using real option theory to value an investment, a density function for the future price of oil is central to the option valuation. The literature on o ..."
Abstract

Cited by 1 (1 self)
 Add to MetaCart
Abstract. For oil related investment appraisal, an accurate description of the evolving uncertainty in the oil price is essential. For example, when using real option theory to value an investment, a density function for the future price of oil is central to the option valuation. The literature on oil pricing offers two views. The arbitrage pricing theory literature for oil suggests geometric Brownian motion and mean reversion models. Empirically driven literature suggests ARMAGARCH models. In addition to reflecting the volatility of the market, the density function of future prices should also incorporate the uncertainty due to price jumps, a common occurrence in the oil market. In this study, the accuracy of density forecasts for up to a year ahead is the major criterion for a comparison of a range of models of oil price behaviour, both those proposed in the literature and following from data analysis. The Kullbach Leibler information criterion is used to measure the accuracy of density forecasts. Using two crude oil price series, Brent and West Texas Intermediate (WTI) representing the US market, we demonstrate that accurate density forecasts are achievable for up to nearly two years ahead using a mixture of two Gaussians innovation process with GARCH and no mean reversion.