## Evaluating Density Forecasts: Forecast Combinations, Model Mixtures, Calibration and Sharpness (2008)

### Cached

### Download Links

Citations: | 12 - 5 self |

### BibTeX

@MISC{Mitchell08evaluatingdensity,

author = {James Mitchell and Kenneth F. Wallis},

title = {Evaluating Density Forecasts: Forecast Combinations, Model Mixtures, Calibration and Sharpness},

year = {2008}

}

### OpenURL

### Abstract

In a recent article Gneiting, Balabdaoui and Raftery (JRSSB, 2007) propose the criterion of sharpness for the evaluation of predictive distributions or density forecasts. They motivate their proposal by an example in which standard evaluation procedures based on probability integral transforms cannot distinguish between the ideal forecast and several competing forecasts. In this paper we show that their example has some unrealistic features from the perspective of the time-series forecasting literature, hence it is an insecure foundation for their argument that existing calibration procedures are inadequate in practice. We present an alternative, more realistic example in which relevant statistical methods, including information-based methods, provide the required discrimination between competing forecasts. We conclude that there is no need for a subsidiary criterion of sharpness.

### Citations

300 | W.,1950 : verification of forecasts expressed in terms of probability - Brier |

289 | Evaluating density forecasts, with applications to financial risk management - Diebold, Gunther, et al. - 1998 |

245 |
Finite mixture distributions
- Everitt, Hand
- 1981
(Show Context)
Citation Context ... can often be of benefit and so constructs the equally-weighted combined forecast 2 2 ( ρ1 −1 σ1 ) ( ρ2 −2 σ2) F = 0.5 N y , + 0.5 N y , , Ct t t which is an example of a finite mixture distribution (=-=Everitt and Hand, 1981-=-). The composite information set for the combined density forecast is identical to the information set of the true forecast density: both contain the same two observations. However the combined foreca... |

195 | The combination of forecasts - Bates - 1969 |

187 |
Statistical theory: The prequential approach
- Dawid
- 1984
(Show Context)
Citation Context ...ly; this ‘has an obvious analogy with the Likelihood Principle, in asserting the irrelevance of hypothetical forecasts that might have been issued in circumstances that did not, in fact, come about’ (=-=Dawid, 1984-=-, p.281). A standard approach is to calculate the probability integral transform values of the outcomes in the forecast distributions, and assessment rests on ‘the question of whether [such] a sequenc... |

170 | Rational decisions - Good - 1952 |

151 | Testing Density Forecasts with Applications to Risk Management - Berkowitz - 2001 |

99 | Contributions to the mathematical theory of evolution - Pearson - 1894 |

78 | Density forecasting: A survey - Tay, Wallis - 2000 |

75 | An omnibus test for univariate and multivariate normality. Discussion paper
- Hansen, Doornik
- 1994
(Show Context)
Citation Context ...ted directly. 3Formal tests of goodness-of-fit include the classical Kolmogorov-Smirnov (KS) and Anderson-Darling (AD) tests for uniformity, together with the Doornik-Hansen (DH) test for normality (=-=Doornik and Hansen, 1994-=-). These are all based on random sampling assumptions, and there are no general results about their performance under autocorrelation. Test of independence can be based on the t p or z t series, as no... |

68 | Comparing density forecasts via weighted likelihood ratio tests - Amisano, Giacomini - 2007 |

63 | 2001: Interpretation of rank histograms for verifying ensemble forecasts - Hamill |

46 | Probabilistic forecasts, calibration and sharpness - Gneiting, Balabdaoui, et al. - 2007 |

43 | Inflation forecast uncertainty - Giordani, Soderlind - 2003 |

36 | Evaluating, comparing and combining density forecasts using the klic with an application to the bank of england and nieser “fan” charts of inflation - Mitchell, Hall - 2005 |

31 | An evaluation of tests of distributional forecasts - Noceti, Smith, et al. - 2003 |

29 | Predictive density evaluation - Corradi, Swanson - 2006 |

25 | Tests for Skewness, Kurtosis, and Normality for Time Series Data - Bai, Ng - 2005 |

25 | On subjective probability forecasting - Sanders - 1963 |

24 | Comparing and Evaluating Bayesian Predictive Distributions of Asset Returns - Geweke, Amisano |

23 | Forecast Combinations,” in Handbook of Economic Forecasting - Timmermann - 2005 |

22 | Optimal Prediction Pools - Geweke, Amisano - 2011 |

21 | Diagnostic checks of non-standard time series models - Smith - 1985 |

15 | Comparing density forecast models - Bao, H, et al. - 2006 |

13 | Combining forecast densities from VARs with uncertain instabilities. Norges Bank, NIESR and RBNZ, working paper - Jore, Mitchell, et al. - 2007 |

12 | Bootstrap conditional distribution tests in the presence of dynamic misspecification
- Corradi, Swanson
- 2003
(Show Context)
Citation Context ...t W t) coincides * with the correct conditional distribution ( ) G x W it satisfies probabilistic calibration – it t t t has uniform PITs – but not necessarily complete calibration (see, for example, =-=Corradi and Swanson, 2006-=-). 2.2. Statistical tests Smith (1985) describes diagnostic checks that can be applied to a range of forecasting models, based on the PIT values −1 transformation, z ( ) t =Φ t p t or on the values gi... |

10 | 2006): “Tests of Conditional Predictive Ability,” Econometrica, forthcoming - Giacomini, White |

5 | Forecasting white noise - Granger - 1983 |

2 | Interpretation of rank histograms for verifying ensemble forecasts - M - 2000 |

1 |
Weather forecasting, Brier score in
- Kroese, Schaafsma
- 2006
(Show Context)
Citation Context ...963), respectively measuring the ‘validity’ and ‘sharpness’ of the forecasts. Subsequent terminology equates validity with calibration or reliability, and sharpness with refinement or resolution (see =-=Kroese and Schaafsma, 2006-=-); both components are functions of the forecastobservation pairs, unlike ‘sharpness’ as redefined by GBR. The logarithmic score for forecast density log S ( x ) log f ( x ) = t . j t jt f jt is defin... |