Scoring rules assess the quality of probabilistic forecasts, by assigning a numerical score based on the predictive distribution and on the event or value that materializes. A scoring rule is proper if the forecaster maximizes the expected score for an observation drawn from the distribution F if he or she issues the probabilistic forecast F, rather than G ̸ = F. It is strictly proper if the maximum is unique. In prediction problems, proper scoring rules encourage the forecaster to make careful assessments and to be honest. In estimation problems, strictly proper scoring rules provide attractive loss and utility functions that can be tailored to the problem at hand. This article reviews and develops the theory of proper scoring rules on general probability spaces, and proposes and discusses examples thereof. Proper scoring rules derive from convex functions and relate to information measures, entropy functions, and Bregman divergences. In the case of categorical variables, we prove a rigorous version of the Savage representation. Examples of scoring rules for probabilistic forecasts in the form of predictive densities include the logarithmic, spherical, pseudospherical, and quadratic scores. The continuous ranked probability score applies to probabilistic forecasts that take the form of predictive cumulative distribution functions. It generalizes the absolute error and forms a special case of a new and very general type of score, the energy score. Like many other scoring rules, the energy score admits a kernel representation in terms of negative definite functions, with links to inequalities of Hoeffding type, in both univariate and multivariate settings. Proper scoring rules for quantile and interval forecasts are also discussed. We relate proper scoring rules to Bayes factors and to cross-validation, and propose a novel form of cross-validation known as random-fold cross-validation. A case study on probabilistic weather forecasts in the North American Pacific Northwest illustrates the importance of propriety. We note optimum score approaches to point and quantile

Technology, February 8-11, 1960 was the subject of Zator Technical Bulletin No. 138. Much of the material of Sections 3.1 to 3.4 rst appeared in Zator Technical Bulletins 138 and 139 of November 1960 and January 1961, respectively. Sections 4.1 and 4.2 are more exact presentations of Zator Technical Bulletins 140 and 141 of April 1961 and April 1962, respectively. 1 In Part II these models are applied to the solution of three problems| prediction of the Bernoulli sequence, extrapolation of a certain kind of Markov chain, and the use of phrase structure grammars for induction. Though some approximations are used, the rst of these problems is treated must rigorously. The result is Laplace's rule of succession. The solution to the second problem uses less certain approximations, but the properties of the solution that are discussed, are fairly independent of these approximations. The third application, using phrase structure grammars, is least exact of the three. Fi

� We survey the literature on prediction mechanisms, including prediction markets and peer prediction systems. We pay particular attention to the design process, highlighting the objectives and properties that are important in the design of good prediction mechanisms. Mechanism design has been described as “inverse game theory. ” Whereas game theorists ask what outcome results from a game, mechanism designers ask what game produces a desired outcome. In this sense, game theorists act like scientists and mechanism designers like engineers. In this article, we survey a number of mechanisms created to elicit predictions, many newly proposed within the last decade. We focus on the engineering questions: How do they work and why? What factors and goals are most important in their

Abstract not found

Scoring rules assess the quality of probabilistic forecasts, by assigning a numerical score based on the predictive distribution and on the event or value that materializes. A scoring rule is proper if it encourages truthful reporting. It is local of order λ if the score depends on the predictive density only through its value and its derivatives of order up to λ at the observation. Previously, only a single local proper scoring rule had been known, namely the logarithmic score, which is local of order λ = 0. Here we introduce the Fisher score, which is a local proper scoring rule of order λ = 2. It relates to the Fisher information in the same way that the logarithmic score relates to the Kullback-Leibler information. The convex cone generated by the logarithmic score and the Fisher score exhausts the class of the local proper scoring rules of order λ ≤ 2, up to equivalence and regularity conditions. In a data example, we use local and non-local proper scoring rules to assess statistically postprocessed ensemble weather forecasts. Finally, we develop a multivariate version of the Fisher score. 1

job market paper

On the domain of Choquet expected utility preferences with risk neutral lottery evaluation and totally monotone capacities, we demonstrate that proper scoring rules do not exist. This implies the non-existence of proper scoring rules for any larger class of preferences (CEU with convex capacities, multiple priors). We also show that if an agent whose behavior conforms to the multiple priors model is faced with a scoring rule for a subjective expected utility agent, she will always announce a probability belonging to her set of priors; moreover, for any prior in the set, there exists such a scoring rule inducing the agent to announce that prior.

Proper scoring rules (PSR) are among the most popular incentivized belief elicitation mechanisms. A well known result is that risk averters facing PSR misreport their beliefs by stating more uniform probabilities. We show that this result does not generalize when i) the PSR payments are increased, ii) the agent has a …nancial stake in the event she is predicting, and iii) the agent can hedge her prediction by taking an additional action. Instead, combining theory and experiment, we …nd that agents distort their reported probabilities in complex, yet mostly predictable manners. We argue that our results have implications for the elicitation of beliefs in most environments of interest to economists, both in academia and in practice. We would especially like to thank Amadou Boly and Alexander Wagner for their help and insights during the course of this research project. We also thank seminar participants at Bergen and the 2009 ESA conference in Tucson for helpful discussions. All remaining errors are ours. The views expressed in this paper are those of the authors and do not necessarily represent the views of the Federal Reserve Bank of New York, or the Federal Reserve System.

Abstract. Decision and prediction markets are designed to determine the likelihood of future events; prediction markets predict what will happen, and decision markets predict the results of a choice, or what would happen. Both allow multiple participants to review and make predictions, and participants are typically scored for improving the accuracy of the market’s prediction. Previous work has demonstrated prediction markets can reward accuracy improvements, as can a single participant informing a decision. We construct and characterize decision markets where all participants are scored for improving the market’s accuracy. These markets require the decision maker always risk taking an action at random, and reducing this risk increases its potential loss. We also relate these decision markets to sets of prediction markets, demonstrating a correspondence between their perfect Bayesian equilibria. 1