Results 1  10
of
111
Strictly Proper Scoring Rules, Prediction, and Estimation
, 2007
"... Scoring rules assess the quality of probabilistic forecasts, by assigning a numerical score based on the predictive distribution and on the event or value that materializes. A scoring rule is proper if the forecaster maximizes the expected score for an observation drawn from the distribution F if he ..."
Abstract

Cited by 143 (17 self)
 Add to MetaCart
Scoring rules assess the quality of probabilistic forecasts, by assigning a numerical score based on the predictive distribution and on the event or value that materializes. A scoring rule is proper if the forecaster maximizes the expected score for an observation drawn from the distribution F if he or she issues the probabilistic forecast F, rather than G ̸ = F. It is strictly proper if the maximum is unique. In prediction problems, proper scoring rules encourage the forecaster to make careful assessments and to be honest. In estimation problems, strictly proper scoring rules provide attractive loss and utility functions that can be tailored to the problem at hand. This article reviews and develops the theory of proper scoring rules on general probability spaces, and proposes and discusses examples thereof. Proper scoring rules derive from convex functions and relate to information measures, entropy functions, and Bregman divergences. In the case of categorical variables, we prove a rigorous version of the Savage representation. Examples of scoring rules for probabilistic forecasts in the form of predictive densities include the logarithmic, spherical, pseudospherical, and quadratic scores. The continuous ranked probability score applies to probabilistic forecasts that take the form of predictive cumulative distribution functions. It generalizes the absolute error and forms a special case of a new and very general type of score, the energy score. Like many other scoring rules, the energy score admits a kernel representation in terms of negative definite functions, with links to inequalities of Hoeffding type, in both univariate and multivariate settings. Proper scoring rules for quantile and interval forecasts are also discussed. We relate proper scoring rules to Bayes factors and to crossvalidation, and propose a novel form of crossvalidation known as randomfold crossvalidation. A case study on probabilistic weather forecasts in the North American Pacific Northwest illustrates the importance of propriety. We note optimum score approaches to point and quantile
The Digitization of WordofMouth: Promise and Challenges of Online Reputation Systems
, 2001
"... Online reputation mechanisms are emerging as a promising alternative to more traditional trust building mechanisms, such as branding and formal contracting, in settings where the latter may be ineffective or prohibitively expensive; a lot of electronic trading communities fall under these categories ..."
Abstract

Cited by 108 (8 self)
 Add to MetaCart
Online reputation mechanisms are emerging as a promising alternative to more traditional trust building mechanisms, such as branding and formal contracting, in settings where the latter may be ineffective or prohibitively expensive; a lot of electronic trading communities fall under these categories. Although a number of commercial websites already employ various forms of reputation mechanisms, rigorous research into their properties is still in its infancy. This fledgling field can benefit from past results in economics and game theory. Moreover, in order to translate the stylized results of game theory into concrete managerial guidance for implementing and participating in effective reputation mechanisms further advances are needed in a number of important areas: First, the design space of such mechanisms needs to be scoped and the effects of different design choices on performance need to be better understood. Second, the economic efficiency of various classes of reputation mechanisms needs to be quantified and compared to that of alternative mechanisms for building trust. Third, the robustness of those mechanisms against boundedly rational players, noisy ratings and strategic manipulation needs to be studied and improved. This paper surveys past results that have been derived in a variety of contexts, but which are relevant as a basis for building online reputation systems, presents two analytical models that illustrate the role of such systems in electronic markets and identifies opportunities for further MS/OR research in this fascinating area.
Logarithmic Market Scoring Rules for Modular Combinatorial Information Aggregation
 Journal of Prediction Markets
, 2002
"... In practice, scoring rules elicit good probability estimates from individuals, while betting markets elicit good consensus estimates from groups. Market scoring rules combine these features, eliciting estimates from individuals or groups, with groups costing no more than individuals. ..."
Abstract

Cited by 72 (5 self)
 Add to MetaCart
In practice, scoring rules elicit good probability estimates from individuals, while betting markets elicit good consensus estimates from groups. Market scoring rules combine these features, eliciting estimates from individuals or groups, with groups costing no more than individuals.
Eliciting Informative Feedback: The PeerPrediction Method
 Management Science
, 2005
"... informs ® doi 10.1287/mnsc.1050.0379 ..."
Interpreting the Predictions of Prediction Markets
"... Participants in prediction markets such as the Iowa Electronic Markets trade allornothing contracts that pay a dollar if and only if specified future events occur. Researchers engaged in empirical study of prediction markets have argued broadly that equilibrium prices of the contracts traded are “ ..."
Abstract

Cited by 42 (2 self)
 Add to MetaCart
Participants in prediction markets such as the Iowa Electronic Markets trade allornothing contracts that pay a dollar if and only if specified future events occur. Researchers engaged in empirical study of prediction markets have argued broadly that equilibrium prices of the contracts traded are “market probabilities ” that the specified events will occur. This paper shows that if traders are riskneutral price takers with heterogenous beliefs, the price of a contract in a prediction market reveals nothing about the dispersion of traders ’ beliefs and partially identifies the central tendency of beliefs. Most persons have beliefs higher than price when price is above 0.5, and most have beliefs lower than price when price is below 0.5. The mean belief of traders lies in an interval whose midpoint is the equilibrium price. These findings persist even if traders use price data to revise their beliefs in plausible ways.
System Identification, Approximation and Complexity
 International Journal of General Systems
, 1977
"... This paper is concerned with establishing broadlybased systemtheoretic foundations and practical techniques for the problem of system identification that are rigorous, intuitively clear and conceptually powerful. A general formulation is first given in which two order relations are postulated on a ..."
Abstract

Cited by 34 (23 self)
 Add to MetaCart
This paper is concerned with establishing broadlybased systemtheoretic foundations and practical techniques for the problem of system identification that are rigorous, intuitively clear and conceptually powerful. A general formulation is first given in which two order relations are postulated on a class of models: a constant one of complexity; and a variable one of approximation induced by an observed behaviour. An admissible model is such that any less complex model is a worse approximation. The general problem of identification is that of finding the admissible subspace of models induced by a given behaviour. It is proved under very general assumptions that, if deterministic models are required then nearly all behaviours require models of nearly maximum complexity. A general theory of approximation between models and behaviour is then developed based on subjective probability concepts and semantic information theory The role of structural constraints such as causality, locality, finite memory, etc., are then discussed as rules of the game. These concepts and results are applied to the specific problem or stochastic automaton, or grammar, inference. Computational results are given to demonstrate that the theory is complete and fully operational. Finally the formulation of identification proposed in this paper is analysed in terms of Klir’s epistemological hierarchy and both are discussed in terms of the rich philosophical literature on the acquisition of knowledge. 1
Combining probability distributions from dependent information sources
 Management Sci
, 1981
"... Your use of the JSTOR archive indicates your acceptance of JSTOR's Terms and Conditions of Use, available at ..."
Abstract

Cited by 33 (1 self)
 Add to MetaCart
Your use of the JSTOR archive indicates your acceptance of JSTOR's Terms and Conditions of Use, available at
Loss Functions for Binary Class Probability Estimation and Classification: Structure and Applications,” manuscript, available at wwwstat.wharton.upenn.edu/~buja
, 2005
"... What are the natural loss functions or fitting criteria for binary class probability estimation? This question has a simple answer: socalled “proper scoring rules”, that is, functions that score probability estimates in view of data in a Fisherconsistent manner. Proper scoring rules comprise most ..."
Abstract

Cited by 33 (1 self)
 Add to MetaCart
What are the natural loss functions or fitting criteria for binary class probability estimation? This question has a simple answer: socalled “proper scoring rules”, that is, functions that score probability estimates in view of data in a Fisherconsistent manner. Proper scoring rules comprise most loss functions currently in use: logloss, squared error loss, boosting loss, and as limiting cases costweighted misclassification losses. Proper scoring rules have a rich structure: • Every proper scoring rules is a mixture (limit of sums) of costweighted misclassification losses. The mixture is specified by a weight function (or measure) that describes which misclassification cost weights are most emphasized by the proper scoring rule. • Proper scoring rules permit Fisher scoring and Iteratively Reweighted LS algorithms for model fitting. The weights are derived from a link function and the above weight function. • Proper scoring rules are in a 11 correspondence with information measures for treebased classification.
The Strategy of Professional Forecasting
 Mimeo, London Business School
, 2003
"... This paper develops and compares two theories of strategic behavior of professional forecasters. The first theory posits that forecasters compete in a forecasting contest with prespecified rules. In equilibrium of a winnertakeall contest, forecasts are excessively differentiated. According to the ..."
Abstract

Cited by 27 (0 self)
 Add to MetaCart
This paper develops and compares two theories of strategic behavior of professional forecasters. The first theory posits that forecasters compete in a forecasting contest with prespecified rules. In equilibrium of a winnertakeall contest, forecasts are excessively differentiated. According to the alternative reputational cheap talk theory, forecasters aim at convincing the market that they are well informed. The market evaluates their forecasting talent on the basis of the forecasts and the realized state. If the market expects forecaster honesty, forecasts are shaded toward the prior mean. With correct market expectations, equilibrium forecasts are imprecise but not shaded.
Boosted classification trees and class probability/quantile estimation
 Journal of Machine Learning Research
, 2006
"... The standard by which binary classifiers are usually judged, misclassification error, assumes equal costs of misclassifying the two classes or, equivalently, classifying at the 1/2 quantile of the conditional class probability function P[y = 1x]. Boosted classification trees are known to perform qu ..."
Abstract

Cited by 23 (4 self)
 Add to MetaCart
The standard by which binary classifiers are usually judged, misclassification error, assumes equal costs of misclassifying the two classes or, equivalently, classifying at the 1/2 quantile of the conditional class probability function P[y = 1x]. Boosted classification trees are known to perform quite well for such problems. In this article we consider the use of standard, offtheshelf boosting for two more general problems: 1) classification with unequal costs or, equivalently, classification at quantiles other than 1/2, and 2) estimation of the conditional class probability function P[y = 1x]. We first examine whether the latter problem, estimation of P[y = 1x], can be solved with LogitBoost, and with AdaBoost when combined with a natural link function. The answer is negative: both approaches are often ineffective because they overfit P[y = 1x] even though they perform well as classifiers. A major negative point of the present article is the disconnect between class probability estimation and classification. Next we consider the practice of over/undersampling of the two classes. We present an algorithm that uses AdaBoost in conjunction with Over/UnderSampling and Jittering of the data (“JOUSBoost”). This algorithm is simple, yet successful, and it preserves the advantage of relative protection against overfitting, but for arbitrary misclassification costs and, equivalently, arbitrary quantile boundaries. We then use collections of classifiers obtained from a grid of quantiles to form estimators of class probabilities. The estimates of the class probabilities compare favorably to those obtained by a variety of methods across both simulated and real data sets.