Abstract

Rank histograms are a tool for evaluating ensemble forecasts. They are useful for determining the reliability of ensemble forecasts and for diagnosing errors in its mean and spread. Rank histograms are generated by repeatedly tallying the rank of the verification (usually, an observation) relative to values from an ensemble sorted from lowest to highest. However, an uncritical use of the rank histogram can lead to misinterpretations of the qualities of that ensemble. For example, a flat rank histogram, ususally taken as a sign of reliability, can still be generated from unreliable ensembles. Similarly, a U-shaped rank histogram, commonly understood as indicating a lack of variability in the ensemble, can also be a sign of conditional bias. It is also shown that flat rank histograms can be generated for some model variables if the variance of the ensemble is correctly specified, yet if covariances between model grid points are improperly specified, rank histograms for combinations of mo...

Citations

485	Deterministic nonperiodic flow - Lorenz - 1963
296	Statistical Inference - Casella, Berger - 1990
149	Verification of forecasts expressed in terms of probabilities - Brier - 1950
78	The ECMWF ensemble prediction system: Methodology and validation - Molteni, Buizza, et al. - 1996
53	Ensemble forecasting at NMC: The generation of perturbations - Toth, Kalnay - 1993
34	A method for producing and evaluating probabilistic forecasts from ensemble model integrations - Anderson - 1996
34	2000: A hybrid ensemble Kalman Filter-3D variational analysis scheme - Hamill, Snyder
34	Statistical methods in the atmospheric sciences, an introduction - Wilks - 1995
27	A model for assessment of weather forecasts - Mason - 1982
26	Evaluation of probabilistic prediction systems - Talagrand, Vautard, et al. - 1997
23	The relative operating characteristic in psychology - SWETS - 1973
21	Disentangling uncertainty and error: On the predictability of nonlinear systems - Smith - 2000
19	2000: A comparison of probabilistic forecasts from bred, singular vector, and perturbed observation ensembles - Snyder, Morss
19	2000: Using initial condition and model physics perturbations in short-range ensemble simulations of mesoscale convective systems - Stensrud, Bao, et al.
17	A Scoring System for Probability Forecasts of Ranked Categories - Epstein - 1969
16	2000: Decomposition on the continuous ranked probability score for ensemble prediction systems - Hersbach
12	Calibrated probabilistic quantitative precipitation forecasts based on the MRF ensemble - Eckel, Walters - 1998
11	The RPN Ensemble Prediction System - Houtekamer, Derome - 1995
10	Ensemble forecasting in the short to medium range: report from a workshop - Hamill, Snyder, et al. - 2000
8	Stochastic simulation of model uncertainties in the ECMWF ensemble prediction system - Buizza, Miller, et al. - 1999
8	2000: Joint medium-range ensembles from U.K. Meteorological Office and ECMWF - Evans, Harrison, et al.
6	Survey of common verification methods in meteorology. Research report 89-5, Environment Canada. Available from Atmospheric Environment - Stanski, Wilson, et al. - 1989
5	Enlightenment in shadows. In Applied nonlinear dynamics and stochastic systems near the millenium - Gilmour, Smith - 1997
5	On the "ranked probability score - Murphy - 1969
4	Evaluation of Eta-RSM ensemble probabilistic precipitation forecasts - Colucci - 1998
4	2000: Ensembles using multiple models and analyses - Richardson - 2000
3	Random and systematic error in NMC’s short-range Eta ensembles - Hamill, Colucci - 1996
2	Medium-range ensembles using both the ECMWF T63 and unified models—An initial report - Harrison, Richardson, et al. - 1995
1	2000: Influence of added observations on analysis and forecast - Morss, Emanuel