Results 1  10
of
11
Strictly Proper Scoring Rules, Prediction, and Estimation
, 2007
"... Scoring rules assess the quality of probabilistic forecasts, by assigning a numerical score based on the predictive distribution and on the event or value that materializes. A scoring rule is proper if the forecaster maximizes the expected score for an observation drawn from the distribution F if he ..."
Abstract

Cited by 357 (27 self)
 Add to MetaCart
Scoring rules assess the quality of probabilistic forecasts, by assigning a numerical score based on the predictive distribution and on the event or value that materializes. A scoring rule is proper if the forecaster maximizes the expected score for an observation drawn from the distribution F if he or she issues the probabilistic forecast F, rather than G ̸ = F. It is strictly proper if the maximum is unique. In prediction problems, proper scoring rules encourage the forecaster to make careful assessments and to be honest. In estimation problems, strictly proper scoring rules provide attractive loss and utility functions that can be tailored to the problem at hand. This article reviews and develops the theory of proper scoring rules on general probability spaces, and proposes and discusses examples thereof. Proper scoring rules derive from convex functions and relate to information measures, entropy functions, and Bregman divergences. In the case of categorical variables, we prove a rigorous version of the Savage representation. Examples of scoring rules for probabilistic forecasts in the form of predictive densities include the logarithmic, spherical, pseudospherical, and quadratic scores. The continuous ranked probability score applies to probabilistic forecasts that take the form of predictive cumulative distribution functions. It generalizes the absolute error and forms a special case of a new and very general type of score, the energy score. Like many other scoring rules, the energy score admits a kernel representation in terms of negative definite functions, with links to inequalities of Hoeffding type, in both univariate and multivariate settings. Proper scoring rules for quantile and interval forecasts are also discussed. We relate proper scoring rules to Bayes factors and to crossvalidation, and propose a novel form of crossvalidation known as randomfold crossvalidation. A case study on probabilistic weather forecasts in the North American Pacific Northwest illustrates the importance of propriety. We note optimum score approaches to point and quantile
Eliciting Informative Feedback: The PeerPrediction Method
 Management Science
, 2005
"... informs ® doi 10.1287/mnsc.1050.0379 ..."
(Show Context)
Composite Multiclass Losses
"... We consider loss functions for multiclass prediction problems. We show when a multiclass loss can be expressed as a “proper composite loss”, which is the composition of a proper loss and a link function. We extend existing results for binary losses to multiclass losses. We determine the stationarity ..."
Abstract

Cited by 19 (8 self)
 Add to MetaCart
We consider loss functions for multiclass prediction problems. We show when a multiclass loss can be expressed as a “proper composite loss”, which is the composition of a proper loss and a link function. We extend existing results for binary losses to multiclass losses. We determine the stationarity condition, Bregman representation, ordersensitivity, existence and uniqueness of the composite representation for multiclass losses. We subsume existing results on “classification calibration ” by relating it to properness and show that the simple integral representation for binary proper losses can not be extended to multiclass losses. 1
Elicitation and evaluation of statistical forecasts
, 2010
"... This paper studies mechanisms for eliciting and evaluating statistical forecasts. Nature draws a state at random from a given state space, according to some distribution p. Prior to Nature’s move, a forecaster, who knows p, provides a prediction for a given statistic of p. The mechanism defines the ..."
Abstract

Cited by 7 (0 self)
 Add to MetaCart
(Show Context)
This paper studies mechanisms for eliciting and evaluating statistical forecasts. Nature draws a state at random from a given state space, according to some distribution p. Prior to Nature’s move, a forecaster, who knows p, provides a prediction for a given statistic of p. The mechanism defines the forecaster’s payoff as a function of the prediction and the subsequently realized state. When the statistic is continuous with a continuum of values, the payoffs that provide strict incentives to the forecaster exist if and only if the statistic partitions the set of distributions into convex subsets. When the underlying state space is finite, and the statistic takes values in a finite set, these payoffs exist if and only if the partition forms a linear crosssection of a Voronoi diagram—that is, if the partition forms a power diagram—a stronger condition than convexity. In both cases, the payoffs can be fully characterized essentially as weighted averages of base functions. Preliminary versions appear in the proceedings of the 9 th and 10 th ACM Conference on Electronic
An Axiomatic Characterization of Wagering Mechanisms ∗
, 2011
"... We construct a budgetbalanced wagering mechanism that flexibly extracts information about event probabilities, as well as the mean, median and other statistics from a group of individuals. We show how our mechanism, called Brier betting mechanism, arises naturally from a modified parimutuel betting ..."
Abstract

Cited by 3 (2 self)
 Add to MetaCart
(Show Context)
We construct a budgetbalanced wagering mechanism that flexibly extracts information about event probabilities, as well as the mean, median and other statistics from a group of individuals. We show how our mechanism, called Brier betting mechanism, arises naturally from a modified parimutuel betting market. We prove that it is essentially the unique wagering mechanism that is anonymous, proportional, sybilproof, and homogeneous. In a Bayesian setting, we find that a slight bias away from truthful reporting may arise under asymmetric information, through the correlation between the total wealth wagered and the event outcome. The bias is driven towards zero as the fraction of any individual’s wealth compared to the group converges towards zero.
Consider a society o...
, 2011
"... We construct a budgetbalanced wagering mechanism that flexibly extracts information about event probabilities, as well as the mean, median and other statistics from a group of individuals. We show how our mechanism, called Brier betting mechanism, arises naturally from a modified parimutuel bettin ..."
Abstract

Cited by 1 (0 self)
 Add to MetaCart
We construct a budgetbalanced wagering mechanism that flexibly extracts information about event probabilities, as well as the mean, median and other statistics from a group of individuals. We show how our mechanism, called Brier betting mechanism, arises naturally from a modified parimutuel betting market. We prove that it is essentially the unique wagering mechanism that is anonymous, proportional, sybilproof, and homogeneous. In a Bayesian setting, we find that a slight bias away from truthful reporting may arise under asymmetric information, through the correlation between the total wealth wagered and the event outcome. The bias is driven towards zero as the fraction of any individual’s wealth compared to the group converges towards zero.
Journal of Machine Learning Research? (2014) n–n+48 Submitted 7/14; Published? Composite Multiclass Losses
"... We consider loss functions for multiclass prediction problems. We show when a multiclass loss can be expressed as a “proper composite loss”, which is the composition of a proper loss and a link function. We extend existing results for binary losses to multiclass losses. We subsume results on “classi ..."
Abstract
 Add to MetaCart
We consider loss functions for multiclass prediction problems. We show when a multiclass loss can be expressed as a “proper composite loss”, which is the composition of a proper loss and a link function. We extend existing results for binary losses to multiclass losses. We subsume results on “classification calibration ” by relating it to properness and show that the simple integral representation for binary proper losses can not be extended to multiclass losses. We determine the stationarity condition, Bregman representation, ordersensitivity, and quasiconvexity of multiclass proper losses. We then characterise the existence and uniqueness of the composite representation for multiclass losses. We show how the composite representation is related to other core properties of a loss: mixability, admissibility and (strong) convexity of multiclass losses which we characterise in terms of the Hessian of the Bayes risk.
2 3 4 5 6 7
"... Proofs subject to correction.Not to be reproducedwithout permission.Contributions to the discussionmust not exceed 400words.Contributions longer than400wordswill be cut by the editor. ..."
Abstract
 Add to MetaCart
(Show Context)
Proofs subject to correction.Not to be reproducedwithout permission.Contributions to the discussionmust not exceed 400words.Contributions longer than400wordswill be cut by the editor.
Elicitability and KnowledgeFree Elicitation with Peer Prediction
"... The elicitation of private information from individuals is crucially important to many realworld tasks. But elicitation is most challenging when it is most useful: when objective (verifiable) truth is inaccessible or unavailable, and there is no “answer key ” available to verify reports. Prior work ..."
Abstract
 Add to MetaCart
(Show Context)
The elicitation of private information from individuals is crucially important to many realworld tasks. But elicitation is most challenging when it is most useful: when objective (verifiable) truth is inaccessible or unavailable, and there is no “answer key ” available to verify reports. Prior work has designed mechanisms that truthfully elicit private information without verification for some restricted set of possible information structures of the participants (i.e. the common prior joint distributions of participants ’ signals). In fact, no mechanism can elicit private information truthfully for all information structures without verification. In this paper, we identify the maximal set of information structures that are truthfully elicitable without verification, and provide a mechanism for such elicitation. This mechanism requires that the designer know the information structure of the participants, which is unavailable in many settings. We then propose a knowledgefree peer prediction mechanism that does not require knowledge of the information structure and can truthfully elicit private information for a set of information structures slightly smaller than the maximal set. This mechanism works for both small and large populations in settings with both binary and nonbinary private signals, and is effective on a strict superset of information structures as compared to prior mechanisms that satisfy these properties.
JID:YJETH AID:4258 /FLA [m1+; v 1.194; Prn:27/06/2014; 9:07] P.1 (128) Available online at www.sciencedirect.com
"... www.elsevier.com/locate/jet An axiomatic characterization of wagering mechanisms ✩ ..."
Abstract
 Add to MetaCart
(Show Context)
www.elsevier.com/locate/jet An axiomatic characterization of wagering mechanisms ✩