Scoring rules assess the quality of probabilistic forecasts, by assigning a numerical score based on the predictive distribution and on the event or value that materializes. A scoring rule is proper if the forecaster maximizes the expected score for an observation drawn from the distribution F if he or she issues the probabilistic forecast F, rather than G ̸ = F. It is strictly proper if the maximum is unique. In prediction problems, proper scoring rules encourage the forecaster to make careful assessments and to be honest. In estimation problems, strictly proper scoring rules provide attractive loss and utility functions that can be tailored to the problem at hand. This article reviews and develops the theory of proper scoring rules on general probability spaces, and proposes and discusses examples thereof. Proper scoring rules derive from convex functions and relate to information measures, entropy functions, and Bregman divergences. In the case of categorical variables, we prove a rigorous version of the Savage representation. Examples of scoring rules for probabilistic forecasts in the form of predictive densities include the logarithmic, spherical, pseudospherical, and quadratic scores. The continuous ranked probability score applies to probabilistic forecasts that take the form of predictive cumulative distribution functions. It generalizes the absolute error and forms a special case of a new and very general type of score, the energy score. Like many other scoring rules, the energy score admits a kernel representation in terms of negative definite functions, with links to inequalities of Hoeffding type, in both univariate and multivariate settings. Proper scoring rules for quantile and interval forecasts are also discussed. We relate proper scoring rules to Bayes factors and to cross-validation, and propose a novel form of cross-validation known as random-fold cross-validation. A case study on probabilistic weather forecasts in the North American Pacific Northwest illustrates the importance of propriety. We note optimum score approaches to point and quantile

informs ® doi 10.1287/mnsc.1050.0379

Abstract not found

The growing literature on learning in games has produced various results on the predictive success of learning theories. These results, however, were based on various methods of comparison. The present paper uses experimental data on a set of four games in order to check on the robustness of rankings among learning rules across measures. We characterise measures along three dimensions: (i) the scoring rule, (ii) the method of comparison, and (iii) the definition of observations and apply all thus defined measures to 12 learning rules.

We investigate the problem of truthfully eliciting an expert’s assessment of a property of a probability distribution, where a property is any real-valued function of the distribution such as mean or variance. We show that not all properties are elicitable; for example, the mean is elicitable and the variance is not. For those that are elicitable, we provide a representation theorem characterizing all payment (or “score”) functions that induce truthful revelation. We also consider the elicitation of sets of properties. We then observe that properties can always be inferred from sets of elicitable properties. This naturally suggests the concept of elicitation complexity; the elicitation complexity of property is the minimal size of such a set implying the property. Finally we discuss applications to prediction markets.

Hanson’s market scoring rules allow us to design a prediction market that still gives useful information even if we have an illiquid market with a limited number of budget-constrained agents. Each agent can “move ” the current price of a market towards their prediction. While this movement still occurs in multi-outcome or multidimensional markets we show that no market-scoring rule, under reasonable conditions, always moves the price directly towards beliefs of the agents. We present a modified version of a market scoring rule for budget-limited traders, and show that it does have the property that, from any starting position, optimal trade by a budget-limited trader will result in the market being moved towards the trader’s true belief. This mechanism also retains several attractive strategic properties of the market scoring rule. 1