Results 11  20
of
124
Probabilistic Wind Speed Forecasting using Ensembles and Bayesian Model Averaging
, 2008
"... the Joint Ensemble Forecasting System (JEFS) under subcontract S0647225 from the University ..."
Abstract

Cited by 14 (8 self)
 Add to MetaCart
the Joint Ensemble Forecasting System (JEFS) under subcontract S0647225 from the University
Selffinanced wagering mechanisms for forecasting
 EC
"... We examine a class of wagering mechanisms designed to elicit truthful predictions from a group of people without requiring any outside subsidy. We propose a number of desirable properties for wagering mechanisms, identifying one mechanism—weightedscore wagering—that satisfies all of the properties. ..."
Abstract

Cited by 11 (5 self)
 Add to MetaCart
We examine a class of wagering mechanisms designed to elicit truthful predictions from a group of people without requiring any outside subsidy. We propose a number of desirable properties for wagering mechanisms, identifying one mechanism—weightedscore wagering—that satisfies all of the properties. Moreover, we show that a singleparameter generalization of weightedscore wagering is the only mechanism that satisfies these properties. We explore some variants of the core mechanism based on practical considerations. Categories and Subject Descriptors
Combining Forecast Densities from VARs with Uncertain Instabilities
, 2008
"... Clark and McCracken (2008) argue that combining realtime point forecasts from VARs of output, prices and interest rates improves point forecast accuracy in the presence of uncertain model instabilities. In this paper, we generalize their approach to consider forecast density combinations and evalua ..."
Abstract

Cited by 11 (8 self)
 Add to MetaCart
Clark and McCracken (2008) argue that combining realtime point forecasts from VARs of output, prices and interest rates improves point forecast accuracy in the presence of uncertain model instabilities. In this paper, we generalize their approach to consider forecast density combinations and evaluations. Whereas Clark and McCracken (2008) show that the point forecast errors from particular equalweight pairwise averages are typically comparable or better than benchmark univariate time series models, we show that neither approach produces accurate realtime forecast densities for recent US data. If greater weight is given to models that allow for the shifts in volatilities associated with the Great Moderation, predictive density accuracy improves substantially.
Prediction Mechanisms That Do Not Incentivize Undesirable Actions
 In WINE
, 2009
"... Abstract. A potential downside of prediction markets is that they may incentivize agents to take undesirable actions in the real world. For example, a prediction market for whether a terrorist attack will happen may incentivize terrorism, and an inhouse prediction market for whether a product will ..."
Abstract

Cited by 11 (1 self)
 Add to MetaCart
Abstract. A potential downside of prediction markets is that they may incentivize agents to take undesirable actions in the real world. For example, a prediction market for whether a terrorist attack will happen may incentivize terrorism, and an inhouse prediction market for whether a product will be successfully released may incentivize sabotage. In this paper, we study principalaligned prediction mechanisms– mechanisms that do not incentivize undesirable actions. We characterize all principalaligned proper scoring rules, and we show an “overpayment” result, which roughly states that with n agents, any prediction mechanism that is principalaligned will, in the worst case, require the principal to pay Θ(n) times as much as a mechanism that is not. We extend our model to allow uncertainties about the principal’s utility and restrictions on agents ’ actions, showing a richer characterization and a similar “overpayment ” result.
Composite Binary Losses
, 2009
"... We study losses for binary classification and class probability estimation and extend the understanding of them from margin losses to general composite losses which are the composition of a proper loss with a link function. We characterise when margin losses can be proper composite losses, explicitl ..."
Abstract

Cited by 11 (8 self)
 Add to MetaCart
We study losses for binary classification and class probability estimation and extend the understanding of them from margin losses to general composite losses which are the composition of a proper loss with a link function. We characterise when margin losses can be proper composite losses, explicitly show how to determine a symmetric loss in full from half of one of its partial losses, introduce an intrinsic parametrisation of composite binary losses and give a complete characterisation of the relationship between proper losses and “classification calibrated ” losses. We also consider the question of the “best ” surrogate binary loss. We introduce a precise notion of “best ” and show there exist situations where two convex surrogate losses are incommensurable. We provide a complete explicit characterisation of the convexity of composite binary losses in terms of the link function and the weight function associated with the proper loss which make up the composite loss. This characterisation suggests new ways of “surrogate tuning”. Finally, in an appendix we present some new algorithmindependent results on the relationship between properness, convexity and robustness to misclassification noise for binary losses and show that all convex proper losses are nonrobust to misclassification noise. 1
Penalized loss functions for Bayesian model comparison
"... The deviance information criterion (DIC) is widely used for Bayesian model comparison, despite the lack of a clear theoretical foundation. DIC is shown to be an approximation to a penalized loss function based on the deviance, with a penalty derived from a crossvalidation argument. This approximati ..."
Abstract

Cited by 10 (0 self)
 Add to MetaCart
The deviance information criterion (DIC) is widely used for Bayesian model comparison, despite the lack of a clear theoretical foundation. DIC is shown to be an approximation to a penalized loss function based on the deviance, with a penalty derived from a crossvalidation argument. This approximation is valid only when the effective number of parameters in the model is much smaller than the number of independent observations. In disease mapping, a typical application of DIC, this assumption does not hold and DIC underpenalizes more complex models. Another deviancebased loss function, derived from the same decisiontheoretic framework, is applied to mixture models, which have previously been considered an unsuitable application for DIC.
Predictive model assessment for count data
, 2007
"... Summary. We discuss tools for the evaluation of probabilistic forecasts and the critique of statistical models for ordered discrete data. Our proposals include a nonrandomized version of the probability integral transform, marginal calibration diagrams and proper scoring rules, such as the predicti ..."
Abstract

Cited by 9 (1 self)
 Add to MetaCart
Summary. We discuss tools for the evaluation of probabilistic forecasts and the critique of statistical models for ordered discrete data. Our proposals include a nonrandomized version of the probability integral transform, marginal calibration diagrams and proper scoring rules, such as the predictive deviance. In case studies, we critique count regression models for patent data, and assess the predictive performance of Bayesian ageperiodcohort models for larynx cancer counts in Germany.
The geometry of proper scoring rules
, 2007
"... A decision problem is defined in terms of an outcome space, an action space and a loss function. Starting from these simple ingredients, we ..."
Abstract

Cited by 9 (0 self)
 Add to MetaCart
A decision problem is defined in terms of an outcome space, an action space and a loss function. Starting from these simple ingredients, we
Composite Multiclass Losses
"... We consider loss functions for multiclass prediction problems. We show when a multiclass loss can be expressed as a “proper composite loss”, which is the composition of a proper loss and a link function. We extend existing results for binary losses to multiclass losses. We determine the stationarity ..."
Abstract

Cited by 9 (5 self)
 Add to MetaCart
We consider loss functions for multiclass prediction problems. We show when a multiclass loss can be expressed as a “proper composite loss”, which is the composition of a proper loss and a link function. We extend existing results for binary losses to multiclass losses. We determine the stationarity condition, Bregman representation, ordersensitivity, existence and uniqueness of the composite representation for multiclass losses. We subsume existing results on “classification calibration ” by relating it to properness and show that the simple integral representation for binary proper losses can not be extended to multiclass losses. 1
A Collaborative Mechanism for Crowdsourcing Prediction Problems
"... Machine Learning competitions such as the Netflix Prize have proven reasonably successful as a method of “crowdsourcing ” prediction tasks. But these competitions have a number of weaknesses, particularly in the incentive structure they create for the participants. We propose a new approach, called ..."
Abstract

Cited by 8 (3 self)
 Add to MetaCart
Machine Learning competitions such as the Netflix Prize have proven reasonably successful as a method of “crowdsourcing ” prediction tasks. But these competitions have a number of weaknesses, particularly in the incentive structure they create for the participants. We propose a new approach, called a Crowdsourced Learning Mechanism, in which participants collaboratively “learn ” a hypothesis for a given prediction task. The approach draws heavily from the concept of a prediction market, where traders bet on the likelihood of a future event. In our framework, the mechanism continues to publish the current hypothesis, and participants can modify this hypothesis by wagering on an update. The critical incentive property is that a participant will profit an amount that scales according to how much her update improves performance on a released test set. 1