Summary. Probabilistic forecasts of continuous variables take the form of predictive densities or predictive cumulative distribution functions. We propose a diagnostic approach to the evaluation of predictive performance that is based on the paradigm of maximizing the sharpness of the predictive distributions subject to calibration. Calibration refers to the statistical consistency between the distributional forecasts and the observations and is a joint property of the predictions and the events that materialize. Sharpness refers to the concentration of the predictive distributions and is a property of the forecasts only. A simple theoretical framework allows us to distinguish between probabilistic calibration, exceedance calibration and marginal calibration. We propose and study tools for checking calibration and sharpness, among them the probability integral transform histogram, marginal calibration plots, the sharpness diagram and proper scoring rules. The diagnostic approach is illustrated by an assessment and ranking of probabilistic forecasts of wind speed at the Stateline wind energy centre in the US Pacific Northwest. In combination with cross-validation or in the time series context, our proposal provides very general, nonparametric alternatives to the use of information criteria for model diagnostics and model selection.

This paper brings together two important but hitherto largely unrelated areas of the forecasting literature, density forecasting and forecast combination. It proposes a simple data-driven approach to direct combination of density forecasts using optimal weights. These optimal weights are those weights that minimize the ‘distance’, as measured by the Kullback-Leibler information criterion, between the forecasted and true but unknown density. We explain how this minimization both can and should be achieved. Comparisons with the optimal combination of point forecasts are made. An application to simple time-series density forecasts and two widely used published density forecasts for U.K. inflation, namely the Bank of England and NIESR “fan” charts, illustrates that combination can but need not always help.

Clark and McCracken (2008) argue that combining real-time point forecasts from VARs of output, prices and interest rates improves point forecast accuracy in the presence of uncertain model instabilities. In this paper, we generalize their approach to consider forecast density combinations and evaluations. Whereas Clark and Mc-Cracken (2008) show that the point forecast errors from particular equal-weight pairwise averages are typically comparable or better than benchmark univariate time series models, we show that neither approach produces accurate real-time forecast densities for recent US data. If greater weight is given to models that allow for the shifts in volatilities associated with the Great Moderation, predictive density accuracy improves substantially.

In this paper we propose a testing procedure for comparing the predictive abilities of possibly misspecified density forecast models. We use the minimum Kullback-Leibler Information Criterion (KLIC) divergence measure to define the distance between the candidate density forecast model and the true model. We use the fact that the inverse-normal transform of the probability integral transforms (PIT) should be IID standard normal as discussed in Berkowitz (2001). To compare the performance of density forecast models in the tails, we use the censored likelihood functions to compute the tail minimum KLIC. The reality check test of White (2000) is then constructed using our distance measure as a loss function. To highlight the merits of our approach, we use the daily S&P500 and NASDAQ return series to conduct an empirical density forecast comparison exercise. A large set of distributions, including some recently proposed flexible distributions to accommodate higher moments, and the ARCH-family volatility specifications are studied. Our empirical findings lend further support of fat-tailedness and skewness of return distributions. In addition, the choice of conditional distribution specification appears to be a much more dominant factor in determining the quality of density forecasts than the choice of volatility specification.

In a recent article Gneiting, Balabdaoui and Raftery (JRSSB, 2007) propose the criterion of sharpness for the evaluation of predictive distributions or density forecasts. They motivate their proposal by an example in which standard evaluation procedures based on probability integral transforms cannot distinguish between the ideal forecast and several competing forecasts. In this paper we show that their example has some unrealistic features from the perspective of the time-series forecasting literature, hence it is an insecure foundation for their argument that existing calibration procedures are inadequate in practice. We present an alternative, more realistic example in which relevant statistical methods, including information-based methods, provide the required discrimination between competing forecasts. We conclude that there is no need for a subsidiary criterion of sharpness.

In this paper we investigate whether and how far density forecasts sensibly can be combined to produce a “better ” pooled density forecast. In so doing we bring together two important but hitherto largely unrelated areas of the forecasting literature in economics, density forecasting and forecast combination. We provide simple Bayesian methods of pooling information across alternative density forecasts. We illustrate the proposed techniques in an application to two widely used published density forecasts for U.K. inflation. We examine whether in practice improved density forecasts for inflation, one year ahead, might have been obtained if one had combined the Bank of England and NIESR density forecasts or “fan charts”. 1

The properties of Pearson's goodness-of-fit test, as used in density forecast evaluation, income distribution analysis and elsewhere, are analysed. The components-of-chi-squared or "Pearson analog" tests of Anderson (1994) are shown to be less generally applicable than was originally claimed. For the case of equiprobable classes, where the general components tests remain valid, a Monte Carlo study shows that tests directed towards skewness and kurtosis may have low power, due to differences between the class boundaries and the intersection points of the distributions being compared. The power of individual component tests can be increased by the use of nonequiprobable classes.

We propose a methodology for producing density forecasts for the output gap in real time using a large number of vector autoregessions in inflation and output gap measures. Density combination utilizes a linear mixture of experts framework to produce potentially non-Gaussian ensemble densities for the unobserved output gap. In our application, we show that data revisions alter substantially our probabilistic assessments of the output gap using a variety of output gap measures derived from univariate detrending filters. The resulting ensemble produces well-calibrated forecast densities for US inflation in real time, in contrast to those from simple univariate autoregressions which ignore the contribution of the output gap. Combining evidence from both linear trends and more flexible univariate detrending filters induces strong multi-modality in the predictive densities for the unobserved output gap. The peaks associated with these two detrending methodologies indicate output gaps of opposite sign for some observations, reflecting the pervasive nature of model uncertainty in our US data.

Combined density nowcasts for quarterly Euro-area GDP growth are produced based on the real-time performance of component models. Components are distinguished by their use of “hard ” and “soft ” indicators. We consider the accuracy of the density nowcasts as within-quarter information on the monthly indicators accumulates. We focus on their ability to anticipate the recent recession probabilistically. We find that the relative utility of “soft ” data increased suddenly during the recession. But as this instability was hard to detect in real-time it helps, when producing nowcasts knowing only one month’s “hard ” data, to weight the different indicators equally. As more monthly “hard ” data arrive, better calibrated densities are obtained by giving a higher weight in the combination to these “hard ” indicators.

Summary. In a recent article Gneiting, Balabdaoui and Raftery (Journal of the Royal Statistical Society B, 2007) propose the criterion of sharpness for the evaluation of predictive distributions or density forecasts. They motivate their proposal by an example in which standard evaluation procedures based on probability integral transforms cannot distinguish between the ideal forecast and several competing forecasts. In this paper we show that their example has some unrealistic features which make it an insecure foundation for their argument that existing calibration procedures are inadequate in practice. We present an alternative, more realistic example in which relevant statistical methods provide the required discrimination between competing forecasts, and argue that there is no need for a subsidiary criterion of sharpness.