## Calibrating Multi-Model Forecast Ensembles with Exchangeable and Missing Members using Bayesian Model Averaging ∗ (2009)

### Cached

### Download Links

Citations: | 7 - 4 self |

### BibTeX

@MISC{Fraley09calibratingmulti-model,

author = {Chris Fraley and Adrian E. Raftery and Tilmann Gneiting and Acknowledgements We and Cliff Mass and Greg Hakim and Jeff Baars and Brian Ancell},

title = {Calibrating Multi-Model Forecast Ensembles with Exchangeable and Missing Members using Bayesian Model Averaging ∗},

year = {2009}

}

### OpenURL

### Abstract

Sloughter for sharing their insights and providing data. This research was sponsored by the National Science Foundation under Joint Ensemble Forecasting System (JEFS) subaward No. S06-47225 with the University Corporation for Atmospheric Research (UCAR), as well as grants No. ATM-0724721 and No. DMS-0706745. Bayesian model averaging (BMA) is a statistical postprocessing technique that generates calibrated and sharp predictive probability density functions (PDFs) from forecast ensembles. It represents the predictive PDF as a weighted average of PDFs centered on the bias-corrected ensemble members, where the weights reflect the relative skill of the individual members over a training period. This work adapts the BMA approach to situations that arise frequently in practice, namely, when one or more of the member forecasts are exchangeable, and when there are missing ensemble members. Exchangeable members differ in random perturbations only, such as the members of bred ensembles, singular vector ensembles, or ensemble Kalman filter systems. Accounting for exchangeability simplifies the BMA approach, in that the BMA weights and the parameters of the component PDFs can be assumed to

### Citations

1077 |
The EM Algorithm and Extensions
- McLachlan, Krishnan
- 1997
(Show Context)
Citation Context ...ns were most likely to materialize. The likelihood function typically cannot be maximized analytically, and so it is maximized using the expectation-maximization (EM) algorithm (Dempster et al. 1977; =-=McLachlan and Krishnan 1997-=-). The EM algorithm is iterative, and alternates between two steps, the expectation or E step, and the maximization or M step. For mixture models, it uses unobserved quantities zist, which can be inte... |

517 | Sequential data assimilation with a nonlinear QG model using Monte Carlo methods to forecast error statistics
- Evensen
- 1994
(Show Context)
Citation Context ...ional Centers for Environmental Prediction and the European Centre for Medium-Range Weather Forecasts (Toth and Kalnay 1993; Molteni et al. 1996), , and the members of ensemble Kalman filter systems (=-=Evensen 1994-=-; Hamill 2005). Accounting for exchangeability facilitates postprocessing, in that the BMA weights and model parameters can be constrained to be equal within each exchangeable group. This simplifies t... |

182 | Strictly proper scoring rules, prediction, and estimation
- Gneiting, Raftery
- 2004
(Show Context)
Citation Context ... weather quantity (Hersbach 2000; Wilks 2006). An alternative form is crps(P, x) = E|X − x| − 1 2 E|X − X′ |, where X and X ′ are independent random variables with distribution P (Grimit et al. 2006; =-=Gneiting and Raftery 2007-=-), and E denotes the expectation operator. The continuous ranked probability score is proper and generalizes the absolute error, to which it reduces in the case of a deterministic forecast. Both the c... |

167 |
Ensemble forecasting at NMC: The generation of perturbations
- Toth, Kalnay
- 1993
(Show Context)
Citation Context ...tures. Examples include members of the bred or singular vector synoptic ensembles used by the National Centers for Environmental Prediction and the European Centre for Medium-Range Weather Forecasts (=-=Toth and Kalnay 1993-=-; Molteni et al. 1996), , and the members of ensemble Kalman filter systems (Evensen 1994; Hamill 2005). Accounting for exchangeability facilitates postprocessing, in that the BMA weights and model pa... |

111 |
Maximum Likelihood for Incomplete Data Via the EM Algorithm
- Dempster, Laird, et al.
- 1977
(Show Context)
Citation Context ...e verifying observations were most likely to materialize. The likelihood function typically cannot be maximized analytically, and so it is maximized using the expectation-maximization (EM) algorithm (=-=Dempster et al. 1977-=-; McLachlan and Krishnan 1997). The EM algorithm is iterative, and alternates between two steps, the expectation or E step, and the maximization or M step. For mixture models, it uses unobserved quant... |

82 | Using Bayesian model averaging to calibrate forecast ensembles
- Raftery, Gneiting, et al.
- 2005
(Show Context)
Citation Context ...recasts over a training period. The initial development of BMA was for weather quantities for which the forecast errors are approximately Gaussian, such as surface temperature and sea-level pressure (=-=Raftery et al. 2005-=-; Wilson et al. 2007). The approach was extended by Sloughter et al. (2007, 2009) to apply to skewed weather variables, such as quantitative precipitation and wind speed. For all variables considered,... |

69 | Interpretation of rank histograms for verifying ensemble forecasts
- Hamill
(Show Context)
Citation Context ...ubject to calibration (Gneiting et al. 2007). Calibration of an ensemble system is assessed by means of the verification rank histogram (Anderson 1996; Hamill and Colucci 1997; Talagrand et al. 1997; =-=Hamill 2001-=-), while calibration of the BMA forecast distributions is evaluated by means of the probability integral transform (PIT) histogram (Raftery et al. 2005; Gneiting et al. 2007). The verification rank hi... |

65 |
Verification of Eta-RSM short-range ensemble forecasts Mon
- Hamill, Colucci
- 1997
(Show Context)
Citation Context ... the sharpness of the predictive distributions subject to calibration (Gneiting et al. 2007). Calibration of an ensemble system is assessed by means of the verification rank histogram (Anderson 1996; =-=Hamill and Colucci 1997-=-; Talagrand et al. 1997; Hamill 2001), while calibration of the BMA forecast distributions is evaluated by means of the probability integral transform (PIT) histogram (Raftery et al. 2005; Gneiting et... |

64 |
L.: A Method for Producing and Evaluating Probabilistic Forecasts from Ensemble Model
- Anderson
- 1996
(Show Context)
Citation Context ... is to maximize the sharpness of the predictive distributions subject to calibration (Gneiting et al. 2007). Calibration of an ensemble system is assessed by means of the verification rank histogram (=-=Anderson 1996-=-; Hamill and Colucci 1997; Talagrand et al. 1997; Hamill 2001), while calibration of the BMA forecast distributions is evaluated by means of the probability integral transform (PIT) histogram (Raftery... |

53 | Probabilistic forecasts, calibration and sharpness
- Gneiting, Balabdaoui, et al.
- 2005
(Show Context)
Citation Context ...semble members, we use the renormalization method as described in Section 4. In probabilistic forecasting, the aim is to maximize the sharpness of the predictive distributions subject to calibration (=-=Gneiting et al. 2007-=-). Calibration of an ensemble system is assessed by means of the verification rank histogram (Anderson 1996; Hamill and Colucci 1997; Talagrand et al. 1997; Hamill 2001), while calibration of the BMA ... |

43 |
The rationale behind the success of multimodel ensembles in seasonal forecasting
- Hagedorn, Doblas-Reyes, et al.
- 2005
(Show Context)
Citation Context ... the proposed extensions. There is considerable recent interest in the use of multi-model ensembles as a means of improving deterministic and probabilistic forecast skill (e.g., Hagedorn et al. 2005; =-=Doblas-Reyes et al. 2005-=-; Park et al. 2008; Weigel et al. 2008). Our extensions allow for the application of the BMA technique to any type and configuration of multi-model ensemble. Contrary to earlier studies, our results s... |

39 |
Initial results of a mesoscale short-range ensemble forecast system over the Pacific Northwest
- Grimit, Mass
- 2002
(Show Context)
Citation Context ...sults for 24-hour surface temperature forecasts from the University of Washington (UW) Mesoscale Ensemble (ME), Ensemble Kalman Filter (EnKF), and ME/EnKF combined systems over the Pacific Northwest (=-=Grimit and Mass 2002-=-; Eckel and Grimit 2005; Dirren et al. 2007; Torn and 4Hakim 2008). Section 4 describes how missing ensemble members can be handled in estimation and prediction, and gives verification results with t... |

37 |
2000: Decomposition of the continuous ranked probability score for ensemble prediction systems
- Hersbach
(Show Context)
Citation Context ...tion, here taking the form of a cumulative distribution function, I is an indicator function, equal to one if the argument is true and equal to zero otherwise, and x is the observed weather quantity (=-=Hersbach 2000-=-; Wilks 2006). An alternative form is crps(P, x) = E|X − x| − 1 2 E|X − X′ |, where X and X ′ are independent random variables with distribution P (Grimit et al. 2006; Gneiting and Raftery 2007), and ... |

34 | Probabilistic quantitative precipitation forecasting using Bayesian model averaging. Monthly Weather Review 135
- Sloughter, Raftery, et al.
- 2007
(Show Context)
Citation Context ... mixture model, bias correction is done for each ensemble member individually by fitting a standard linear regression model to the training data (Raftery et al. 2005). For quantitative precipitation (=-=Sloughter et al. 2007-=-), a logistic regression model for the probability of precipitation is fit for each ensemble member. These initial procedures allow for straightforward adaptation when ensemble members are exchangeabl... |

31 | 2005: Ensemble-based atmospheric data assimilation: A tutorial
- Hamill
(Show Context)
Citation Context ...for Environmental Prediction and the European Centre for Medium-Range Weather Forecasts (Toth and Kalnay 1993; Molteni et al. 1996), , and the members of ensemble Kalman filter systems (Evensen 1994; =-=Hamill 2005-=-). Accounting for exchangeability facilitates postprocessing, in that the BMA weights and model parameters can be constrained to be equal within each exchangeable group. This simplifies the BMA approa... |

29 |
A comparison of the ecmwf, msc and ncep global ensemble prediction systems
- Buizza, Houtekamer, et al.
- 2005
(Show Context)
Citation Context ... which some or all of the ensemble members are statistically indistinguishable, differing only in some random perturbations. Most of the synoptic ensembles that are currently in use are of this type (=-=Buizza et al. 2005-=-; Park et al. 2008). In these cases, members that are statistically indistinguishable should be treated as exchangeable, and thus should have equal BMA weight and equal BMA parameter values. The Gauss... |

15 | Probabilistic wind speed forecasting using ensembles and Bayesian model averaging
- Sloughter, Gneiting, et al.
- 2010
(Show Context)
Citation Context ...nts that the BMA mean or bias parameters and the BMA weights be equal, as described by Wilson et al. (2007). In the gamma models for quantitative precipitation (Sloughter et al. 2007) and wind speed (=-=Sloughter et al. 2009-=-), the variance parameters need to be constrained to be equal within exchangeable groups of ensemble members. 3.1 Estimation with exchangeable ensemble members Consider an ensemble of size m, where th... |

14 | 2008: Performance characteristics of a pseudo-operational ensemble Kalman
- Torn, Hakim
(Show Context)
Citation Context ...rsity of Washington (UW) Mesoscale Ensemble (ME) (Grimit and Mass 2002; Eckel and Mass 2005) and the recently developed UW Ensemble Kalman Filter (EnKF) systems (Torn et al. 2006; Dirren et al. 2007; =-=Torn and Hakim 2008-=-) over the Pacific Northwest and adjacent areas of the Pacific Ocean. The version of the UW ME used here is an eight-member multi-analysis ensemble based on the fifth-generation Pennsylvania State Uni... |

13 |
TIGGE: preliminary results on comparing and combining ensembles
- Park, Buizza, et al.
- 2008
(Show Context)
Citation Context ...There is considerable recent interest in the use of multi-model ensembles as a means of improving deterministic and probabilistic forecast skill (e.g., Hagedorn et al. 2005; Doblas-Reyes et al. 2005; =-=Park et al. 2008-=-; Weigel et al. 2008). Our extensions allow for the application of the BMA technique to any type and configuration of multi-model ensemble. Contrary to earlier studies, our results show that statistic... |

13 |
Analysis of Incomplete Multivariate Data by Simulation
- Schafer
- 1997
(Show Context)
Citation Context ...rimented with the median, with nearly identical results.) Conditional Mean: Missing members are replaced using a standard procedure for missing data handling that is known as single imputation (e.g., =-=Schafer 1997-=-), as implemented in the R package norm. For details see Appendix C. 16Renormalized: The model is reduced to the terms for the non-missing forecasts, with the BMA weights renormalized to sum to 1. A ... |

12 | 2007a: Calibrated surface temperature forecasts from the Canadian ensemble prediction system using Bayesian model averaging - Wilson, Beauregard, et al. |

11 |
The continuous ranked probability score for circular variables and its application to mesoscale forecast ensemble verification
- EP, Gneiting, et al.
- 2006
(Show Context)
Citation Context ...nd x is the observed weather quantity (Hersbach 2000; Wilks 2006). An alternative form is crps(P, x) = E|X − x| − 1 2 E|X − X′ |, where X and X ′ are independent random variables with distribution P (=-=Grimit et al. 2006-=-; Gneiting and Raftery 2007), and E denotes the expectation operator. The continuous ranked probability score is proper and generalizes the absolute error, to which it reduces in the case of a determi... |

9 |
The ecmwf ensemble system: Methodology and validation
- Molteni, Buizza, et al.
- 1996
(Show Context)
Citation Context ...e members of the bred or singular vector synoptic ensembles used by the National Centers for Environmental Prediction and the European Centre for Medium-Range Weather Forecasts (Toth and Kalnay 1993; =-=Molteni et al. 1996-=-), , and the members of ensemble Kalman filter systems (Evensen 1994; Hamill 2005). Accounting for exchangeability facilitates postprocessing, in that the BMA weights and model parameters can be const... |

8 | 2007: A data assimilation case study using a limited-area ensemble Kalman filter
- Dirren, Torn, et al.
(Show Context)
Citation Context ...ts from the University of Washington (UW) Mesoscale Ensemble (ME), Ensemble Kalman Filter (EnKF), and ME/EnKF combined systems over the Pacific Northwest (Grimit and Mass 2002; Eckel and Grimit 2005; =-=Dirren et al. 2007-=-; Torn and 4Hakim 2008). Section 4 describes how missing ensemble members can be handled in estimation and prediction, and gives verification results with these methods for 48-hour surface temperatur... |

8 | 2003: Effective Mesoscale, Short-Range Ensemble Forecasting
- Eckel
(Show Context)
Citation Context ...n as z (k+1) ijst = ∑ I p=1 w (k) i gi(yst|fijst, θ (k) i ) ∑mp q=1 w(k) p gp(yst|fpqst, θ (k) p ) . (6) 7Table 1: Composition of the eight member University of Washington Mesoscale Ensemble (UW ME; =-=Eckel and Mass 2005-=-), with member acronyms, and organizational and synoptic model sources for initial and lateral boundary conditions. The organizational sources are the United States National Centers for Environmental ... |

8 | Boundary conditions for limited-area ensemble Kalman filters
- Torn, Hakim, et al.
- 2006
(Show Context)
Citation Context ...xperiment below, we use both the University of Washington (UW) Mesoscale Ensemble (ME) (Grimit and Mass 2002; Eckel and Mass 2005) and the recently developed UW Ensemble Kalman Filter (EnKF) systems (=-=Torn et al. 2006-=-; Dirren et al. 2007; Torn and Hakim 2008) over the Pacific Northwest and adjacent areas of the Pacific Ocean. The version of the UW ME used here is an eight-member multi-analysis ensemble based on th... |

6 |
The THORPEX interactive grand global ensemble (TIGGE
- Bougeault, Fuentes, et al.
- 2010
(Show Context)
Citation Context ...usion which suggests BMA postprocessing for other types of multimodel ensembles, including but not limited to the DEMETER (Hagedorn et al. 2005; Doblas-Reyes et al. 2005) and TIGGE (Park et al. 2008; =-=Bougeault et al. 2009-=-) systems. A Verification scores Scoring rules provide summary measures of predictive performance that address calibration and sharpness simultaneously. A particularly attractive scoring rule for prob... |

6 |
2008: Can multimodel combination really enhance the prediction skill of probabilistic ensemble forecasts
- Weigel, Liniger, et al.
(Show Context)
Citation Context ...ble recent interest in the use of multi-model ensembles as a means of improving deterministic and probabilistic forecast skill (e.g., Hagedorn et al. 2005; Doblas-Reyes et al. 2005; Park et al. 2008; =-=Weigel et al. 2008-=-). Our extensions allow for the application of the BMA technique to any type and configuration of multi-model ensemble. Contrary to earlier studies, our results show that statistically postprocessed m... |

5 |
Strategies in naturalistic decision-making: A cognitive task analysis of naval weather forecasting
- Joslyn, Jones
- 2008
(Show Context)
Citation Context ... in line with how operational weather forecasters often work, by selecting one or a small number of ‘best’ forecasts from a potentially large number available, based on recent predictive performance (=-=Joslyn and Jones 2008-=-). For weather variables such as temperature and sea level pressure, which have approximately Gaussian errors, the component PDFs can be taken to be normal distributions centered at bias-corrected ens... |

4 | 2007: Comments on “Calibrated Surface Temperature Forecasts from the Canadian Ensemble Prediction System Using Bayesian Model Averaging - Hamill |

3 | 2007: EnsembleBMA: An R package for probabilistic forecasting using ensembles and Bayesian model averaging - Fraley, Raftery, et al. |

2 | 2005: The rationale behind the success of multi-model ensembles in seasonal forecasting – I. Basic concept - R, Palmer |

2 |
2005: SRNWP-PEPS: A regional multi-model ensemble in Europe
- Heizenreder, Trepte, et al.
(Show Context)
Citation Context ...ach. It is possible that a different conclusion would be reached if there are higher proportions of missing members, or in situations such as the European Poor Man’s Ensemble Prediction System (PEPS; =-=Heizenreder et al. 2005-=-), in which the ensemble members have overlapping but distinct model domains. 5 Discussion We have shown that the BMA approach to the statistical postprocessing of forecast ensembles can be extended t... |

2 | 2009: Replacing missing data for ensemble systems - McCandless, Haupt, et al. - 1997 |

1 | Evaluation of probabilistic prediction systems - O, Strauss - 1997 |