## [Extended Abstract]

### Cached

### Download Links

### BibTeX

@MISC{Lambert_[extendedabstract],

author = {Nicolas Lambert and David M. Pennock and Yoav Shoham},

title = {[Extended Abstract]},

year = {}

}

### OpenURL

### Abstract

We investigate the problem of truthfully eliciting an expert’s assessment of a property of a probability distribution, where a property is any real-valued function of the distribution such as mean or variance. We show that not all properties are elicitable; for example, the mean is elicitable and the variance is not. For those that are elicitable, we provide a representation theorem characterizing all payment (or “score”) functions that induce truthful revelation. We also consider the elicitation of sets of properties. We then observe that properties can always be inferred from sets of elicitable properties. This naturally suggests the concept of elicitation complexity; the elicitation complexity of property is the minimal size of such a set implying the property. Finally we discuss applications to prediction markets.

### Citations

187 | Strictly proper scoring rules, prediction, and estimation
- Gneiting, Raftery
(Show Context)
Citation Context ...prior publications on truthful elicitation of specific properties. Savage’s [14] original article proposes an adaptation of scoring rules to elicit expectations. Cervera, Munoz, Gneiting, and Raftery =-=[7, 3]-=- suggest scores that can elicit the quantiles for continuous outcomes. Our results are considerably more general, and the highlights are as follows. A property is any real-valued function of a probabi... |

174 | Rational Decisions - Good - 1952 |

153 |
Elicitation of personal probabilities and expectations
- Savage
- 1971
(Show Context)
Citation Context ...en it is possible to induce a truthful estimate of the property with a given dimensionality of reports. We are aware of two prior publications on truthful elicitation of specific properties. Savage’s =-=[14]-=- original article proposes an adaptation of scoring rules to elicit expectations. Cervera, Munoz, Gneiting, and Raftery [7, 3] suggest scores that can elicit the quantiles for continuous outcomes. Our... |

115 | Combinatorial information market design
- Hanson
(Show Context)
Citation Context ...Sports, Betfair, the Iowa Electronic Markets, or the Google internal prediction market – and markets operated by automatic market makers, such as Hanson’s logarithmic market scoring rule market maker =-=[9]-=- – used in YooNew 5 , Inkling Markets 6 , and in Microsoft’s internal prediction markets. 6.1 Markets with continuous double auction A Continuous Double Auction (CDA) attempts to match, at any time, o... |

108 | Prediction Markets
- Wolfers, Zitzewitz
- 2004
(Show Context)
Citation Context ... APPLICATION TO PREDICTION MARKETS Prediction markets are markets designed for the purpose of estimating probabilistic information. Empirical evidence suggests that they are powerful prediction tools =-=[5, 20]-=-, and they are now used regularly in industry. In prediction markets, participants trade securities whose payoffs are contingent upon the realization of uncertain events. Relying on the efficient mark... |

37 |
A general method for comparing probability assessors
- Schervish
- 1989
(Show Context)
Citation Context ...arbitrary but well structured. One of the first characterizations goes back to Shuford et al. [16], who considers eliciting a distribution on binary events. The idea is pushed further by Schervish in =-=[15]-=-, who shows that there exists a one-to-one correspondence between proper scoring rules (for probability of a binary event) and non-negative measures on [0, 1]. Recently, Buja et al. [2] propose a taxo... |

36 | Loss functions for binary class probability estimation: structure and applications
- Buja, Stuetzle, et al.
- 2005
(Show Context)
Citation Context ... Schervish in [15], who shows that there exists a one-to-one correspondence between proper scoring rules (for probability of a binary event) and non-negative measures on [0, 1]. Recently, Buja et al. =-=[2]-=- propose a taxonomy of scoring rules based on their Schervish measure, and use it in a statistical learning setting. We obtain a generalization the results of Schervish [15]. We show that, for an elic... |

34 |
Verification of Forecasts Expressed
- Brier
(Show Context)
Citation Context ... The idea is pushed further by Schervish in [15], who shows that there exists a one-to-one correspondence between proper scoring rules (for probability of a binary event) and non-negative measures on =-=[0, 1]-=-. Recently, Buja et al. [2] propose a taxonomy of scoring rules based on their Schervish measure, and use it in a statistical learning setting. We obtain a generalization the results of Schervish [15]... |

33 |
Anatomy of an Experimental
- Forsythe, Nelson, et al.
- 1992
(Show Context)
Citation Context ... APPLICATION TO PREDICTION MARKETS Prediction markets are markets designed for the purpose of estimating probabilistic information. Empirical evidence suggests that they are powerful prediction tools =-=[5, 20]-=-, and they are now used regularly in industry. In prediction markets, participants trade securities whose payoffs are contingent upon the realization of uncertain events. Relying on the efficient mark... |

27 |
Probabilistic prediction in patient management and clinical trials
- Spiegelhalter
- 1986
(Show Context)
Citation Context ... Copyright 2008 ACM 978-1-60558-169-9/08/07 ...$5.00. 8, 14], scoring rules have been extensively studied over the past five decades, validated experimentally [12] and applied in a variety of domains =-=[11, 17, 13]-=-. Winkler [19] gives a partial summary of the literature. Scoring rules elicit probabilities directly. That is, the expert’s report takes the form of a full probability distribution. This poses practi... |

26 |
Scoring rules and the evaluation of probabilities
- Winkler
- 1996
(Show Context)
Citation Context ...8-1-60558-169-9/08/07 ...$5.00. 8, 14], scoring rules have been extensively studied over the past five decades, validated experimentally [12] and applied in a variety of domains [11, 17, 13]. Winkler =-=[19]-=- gives a partial summary of the literature. Scoring rules elicit probabilities directly. That is, the expert’s report takes the form of a full probability distribution. This poses practical difficulti... |

23 |
Proper scores for probability forecasters
- Hendrickson, Buehler
- 1971
(Show Context)
Citation Context ...ributions ∆(Ω) or those assigning positive probability to each outcome. For the remaining of this paper we will make two assumptions. The first concerns the shape of the possible probabilities: as in =-=[10, 14]-=- we will assume convex domains D. The second concerns the property itself: we will restrict our attention to nice properties Γ satisfying the following conditions: (1) Γ is continuous, and (2) Γ is no... |

22 | 2005), “Computation in a distributed information market
- Feigenbaum, Fortnow, et al.
(Show Context)
Citation Context ...mate of Γ. Thus only properties that correspond to linear constraints may be revealed by the market. It is worth mentioning the manifest connection between our findings and those of Feigenbaum et al. =-=[4]-=-. Feigenbaum et al. model prediction markets as Shapley-Shubik games in which traders receive private binary signals. They show that, under Aumann’s information partition model with common prior, pred... |

18 |
Effective Scoring Rules for Probabilistic Forecasts
- Friedman
- 1983
(Show Context)
Citation Context ...report is closer to the true value of the property than that of another forecaster has higher expected reward. This idea originated from Staël von Holstein [18] and was later investigated by Friedman =-=[6]-=-. We say that a score function whose expected value increases with accuracy is accuracy-rewarding. Accuracyrewarding score functions are also strictly proper. Definition 4. A score function s for a di... |

18 |
E.H.Massengill, "Admissible probability measurement procedures
- Shuford
- 1966
(Show Context)
Citation Context ...r), which concludes the proof. 3.2 Score function representations Strictly proper score functions are not arbitrary but well structured. One of the first characterizations goes back to Shuford et al. =-=[16]-=-, who considers eliciting a distribution on binary events. The idea is pushed further by Schervish in [15], who shows that there exists a one-to-one correspondence between proper scoring rules (for pr... |

13 |
Subjective Probabilities and Scoring Rules: Experimental Evidence
- Nelson, Bessler
(Show Context)
Citation Context ... July 8–12, 2008, Chicago, Illinois, USA. Copyright 2008 ACM 978-1-60558-169-9/08/07 ...$5.00. 8, 14], scoring rules have been extensively studied over the past five decades, validated experimentally =-=[12]-=- and applied in a variety of domains [11, 17, 13]. Winkler [19] gives a partial summary of the literature. Scoring rules elicit probabilities directly. That is, the expert’s report takes the form of a... |

7 |
Subjective probabilities and short-term economic forecasts: An empirical investigation
- O’Carroll
- 1977
(Show Context)
Citation Context ... Copyright 2008 ACM 978-1-60558-169-9/08/07 ...$5.00. 8, 14], scoring rules have been extensively studied over the past five decades, validated experimentally [12] and applied in a variety of domains =-=[11, 17, 13]-=-. Winkler [19] gives a partial summary of the literature. Scoring rules elicit probabilities directly. That is, the expert’s report takes the form of a full probability distribution. This poses practi... |

5 |
Proper scoring rules for fractiles
- Cervera, Munoz
- 1996
(Show Context)
Citation Context ...prior publications on truthful elicitation of specific properties. Savage’s [14] original article proposes an adaptation of scoring rules to elicit expectations. Cervera, Munoz, Gneiting, and Raftery =-=[7, 3]-=- suggest scores that can elicit the quantiles for continuous outcomes. Our results are considerably more general, and the highlights are as follows. A property is any real-valued function of a probabi... |

4 |
Probability Forecasting in Meterology
- Murphy, Winkler
- 1984
(Show Context)
Citation Context ... Copyright 2008 ACM 978-1-60558-169-9/08/07 ...$5.00. 8, 14], scoring rules have been extensively studied over the past five decades, validated experimentally [12] and applied in a variety of domains =-=[11, 17, 13]-=-. Winkler [19] gives a partial summary of the literature. Scoring rules elicit probabilities directly. That is, the expert’s report takes the form of a full probability distribution. This poses practi... |

1 |
von Holstein. A Family of Strictly Proper Scoring Rules Which Are Sensitive to Distance
- Staël
- 1970
(Show Context)
Citation Context ...It may be desirable that a forecaster whose report is closer to the true value of the property than that of another forecaster has higher expected reward. This idea originated from Staël von Holstein =-=[18]-=- and was later investigated by Friedman [6]. We say that a score function whose expected value increases with accuracy is accuracy-rewarding. Accuracyrewarding score functions are also strictly proper... |