Results 1  10
of
13
Strictly Proper Scoring Rules, Prediction, and Estimation
, 2007
"... Scoring rules assess the quality of probabilistic forecasts, by assigning a numerical score based on the predictive distribution and on the event or value that materializes. A scoring rule is proper if the forecaster maximizes the expected score for an observation drawn from the distribution F if he ..."
Abstract

Cited by 143 (17 self)
 Add to MetaCart
Scoring rules assess the quality of probabilistic forecasts, by assigning a numerical score based on the predictive distribution and on the event or value that materializes. A scoring rule is proper if the forecaster maximizes the expected score for an observation drawn from the distribution F if he or she issues the probabilistic forecast F, rather than G ̸ = F. It is strictly proper if the maximum is unique. In prediction problems, proper scoring rules encourage the forecaster to make careful assessments and to be honest. In estimation problems, strictly proper scoring rules provide attractive loss and utility functions that can be tailored to the problem at hand. This article reviews and develops the theory of proper scoring rules on general probability spaces, and proposes and discusses examples thereof. Proper scoring rules derive from convex functions and relate to information measures, entropy functions, and Bregman divergences. In the case of categorical variables, we prove a rigorous version of the Savage representation. Examples of scoring rules for probabilistic forecasts in the form of predictive densities include the logarithmic, spherical, pseudospherical, and quadratic scores. The continuous ranked probability score applies to probabilistic forecasts that take the form of predictive cumulative distribution functions. It generalizes the absolute error and forms a special case of a new and very general type of score, the energy score. Like many other scoring rules, the energy score admits a kernel representation in terms of negative definite functions, with links to inequalities of Hoeffding type, in both univariate and multivariate settings. Proper scoring rules for quantile and interval forecasts are also discussed. We relate proper scoring rules to Bayes factors and to crossvalidation, and propose a novel form of crossvalidation known as randomfold crossvalidation. A case study on probabilistic weather forecasts in the North American Pacific Northwest illustrates the importance of propriety. We note optimum score approaches to point and quantile
Eliciting Informative Feedback: The PeerPrediction Method
 Management Science
, 2005
"... informs ® doi 10.1287/mnsc.1050.0379 ..."
Robustness of preferences among auction institutions
 Economic Inquiry
, 2008
"... This study examines bidder preferences between alternative auction institutions. In particular we seek to experimentally characterize the degree to which bidders prefer an ascending auction over a sealed bid auction. We find very strong ceteris paribus preferences for the ascending institution with ..."
Abstract

Cited by 13 (1 self)
 Add to MetaCart
This study examines bidder preferences between alternative auction institutions. In particular we seek to experimentally characterize the degree to which bidders prefer an ascending auction over a sealed bid auction. We find very strong ceteris paribus preferences for the ascending institution with bidders choosing it overwhelmingly often when entry prices for the two auctions are the same. When the entry prices of the two auctions differ, many subjects can be shown to be willing to pay far more to enter the ascending auction than is explainable by their risk attitudes when accounting for their expectations about the risk preferences of their opponents.
Peer Prediction without a Common Prior
 In Proceedings of the 13th ACM Conference on Electronic Commerce (EC 2012). ACM
"... Reputation mechanisms at online opinion forums, such as Amazon Reviews, elicit ratings from users about their experience with different products. Crowdsourcing applications, such as image tagging on Amazon Mechanical Turk, elicit votes from users as to whether or not a job was duly completed. An imp ..."
Abstract

Cited by 4 (1 self)
 Add to MetaCart
Reputation mechanisms at online opinion forums, such as Amazon Reviews, elicit ratings from users about their experience with different products. Crowdsourcing applications, such as image tagging on Amazon Mechanical Turk, elicit votes from users as to whether or not a job was duly completed. An important property in both settings is that the feedback received from users (agents) is truthful. The peer prediction method introduced by Miller et al. [2005] is a prominent theoretical mechanism for the truthful elicitation of reports. However, a significant obstacle to its application is that it critically depends on the assumption of a common prior amongst both the agents and the mechanism. In this paper, we develop a peer prediction mechanism for settings where the agents hold subjective and private beliefs about the state of the world and the likelihood of a positive signal given a particular state. Our shadow peer prediction mechanism exploits temporal structure in order to elicit two reports, a belief report and then a signal report, and it provides strict incentives for truthful reporting as long as the effect an agent’s signal has on her posterior belief is bounded away from zero. Alternatively, this technical requirement on beliefs can be dispensed with by a modification in which the second report is a belief report rather than a signal report.
Elicitation and evaluation of statistical forecasts
, 2010
"... This paper studies mechanisms for eliciting and evaluating statistical forecasts. Nature draws a state at random from a given state space, according to some distribution p. Prior to Nature’s move, a forecaster, who knows p, provides a prediction for a given statistic of p. The mechanism defines the ..."
Abstract

Cited by 4 (0 self)
 Add to MetaCart
This paper studies mechanisms for eliciting and evaluating statistical forecasts. Nature draws a state at random from a given state space, according to some distribution p. Prior to Nature’s move, a forecaster, who knows p, provides a prediction for a given statistic of p. The mechanism defines the forecaster’s payoff as a function of the prediction and the subsequently realized state. When the statistic is continuous with a continuum of values, the payoffs that provide strict incentives to the forecaster exist if and only if the statistic partitions the set of distributions into convex subsets. When the underlying state space is finite, and the statistic takes values in a finite set, these payoffs exist if and only if the partition forms a linear crosssection of a Voronoi diagram—that is, if the partition forms a power diagram—a stronger condition than convexity. In both cases, the payoffs can be fully characterized essentially as weighted averages of base functions. Preliminary versions appear in the proceedings of the 9 th and 10 th ACM Conference on Electronic
On the Measurement of the Predictive Success of Learning Theories in Repeated Games
, 2001
"... The growing literature on learning in games has produced various results on the predictive success of learning theories. These results, however, were based on various methods of comparison. The present paper uses experimental data on a set of four games in order to check on the robustness of rank ..."
Abstract

Cited by 1 (0 self)
 Add to MetaCart
The growing literature on learning in games has produced various results on the predictive success of learning theories. These results, however, were based on various methods of comparison. The present paper uses experimental data on a set of four games in order to check on the robustness of rankings among learning rules across measures. We characterise measures along three dimensions: (i) the scoring rule, (ii) the method of comparison, and (iii) the definition of observations and apply all thus defined measures to 12 learning rules.
An Axiomatic Characterization of Wagering Mechanisms ∗
, 2011
"... We construct a budgetbalanced wagering mechanism that flexibly extracts information about event probabilities, as well as the mean, median and other statistics from a group of individuals. We show how our mechanism, called Brier betting mechanism, arises naturally from a modified parimutuel betting ..."
Abstract

Cited by 1 (0 self)
 Add to MetaCart
We construct a budgetbalanced wagering mechanism that flexibly extracts information about event probabilities, as well as the mean, median and other statistics from a group of individuals. We show how our mechanism, called Brier betting mechanism, arises naturally from a modified parimutuel betting market. We prove that it is essentially the unique wagering mechanism that is anonymous, proportional, sybilproof, and homogeneous. In a Bayesian setting, we find that a slight bias away from truthful reporting may arise under asymmetric information, through the correlation between the total wealth wagered and the event outcome. The bias is driven towards zero as the fraction of any individual’s wealth compared to the group converges towards zero.
[Extended Abstract]
"... We investigate the problem of truthfully eliciting an expert’s assessment of a property of a probability distribution, where a property is any realvalued function of the distribution such as mean or variance. We show that not all properties are elicitable; for example, the mean is elicitable and th ..."
Abstract
 Add to MetaCart
We investigate the problem of truthfully eliciting an expert’s assessment of a property of a probability distribution, where a property is any realvalued function of the distribution such as mean or variance. We show that not all properties are elicitable; for example, the mean is elicitable and the variance is not. For those that are elicitable, we provide a representation theorem characterizing all payment (or “score”) functions that induce truthful revelation. We also consider the elicitation of sets of properties. We then observe that properties can always be inferred from sets of elicitable properties. This naturally suggests the concept of elicitation complexity; the elicitation complexity of property is the minimal size of such a set implying the property. Finally we discuss applications to prediction markets.
Multioutcome and Multidimensional Market Scoring Rules (Manuscript)
, 2012
"... Hanson’s market scoring rules allow us to design a prediction market that still gives useful information even if we have an illiquid market with a limited number of budgetconstrained agents. Each agent can “move ” the current price of a market towards their prediction. While this movement still occ ..."
Abstract
 Add to MetaCart
Hanson’s market scoring rules allow us to design a prediction market that still gives useful information even if we have an illiquid market with a limited number of budgetconstrained agents. Each agent can “move ” the current price of a market towards their prediction. While this movement still occurs in multioutcome or multidimensional markets we show that no marketscoring rule, under reasonable conditions, always moves the price directly towards beliefs of the agents. We present a modified version of a market scoring rule for budgetlimited traders, and show that it does have the property that, from any starting position, optimal trade by a budgetlimited trader will result in the market being moved towards the trader’s true belief. This mechanism also retains several attractive strategic properties of the market scoring rule. 1