## Defensive Forecasting

Citations: | 13 - 12 self |

### BibTeX

@MISC{Vovk_defensiveforecasting,

author = {Vladimir Vovk Vovk},

title = {Defensive Forecasting},

year = {}

}

### OpenURL

### Abstract

We consider how to make probability forecasts of binary labels. Our main mathematical result is that for any continuous gambling strategy used for detecting disagreement between the forecasts and the actual labels, there exists a forecasting strategy whose forecasts are ideal as far as this gambling strategy is concerned. A forecasting strategy obtained in this way from a gambling strategy demonstrating a strong law of large numbers is simplified and studied empirically.

### Citations

8980 | Statistical Learning Theory
- Vapnik
- 1998
(Show Context)
Citation Context ... squared product ([0, 1] × X) 2 . There are standard ways of constructing such Mercer kernels from Mercer kernels on [0, 1] 2 and X 2 (see, e.g., the description of tensor products and direct sums in =-=[13, 11]-=-). For Sn to be continuous, we have to require that K be forecast-continuous in the following sense: for all x ∈ X and all (p ′ , x ′ ) ∈ [0, 1] × X, K((p, x), (p ′ , x ′ )) is continuous as a functio... |

2028 | Learning with Kernels
- Scholkopf, Smola
- 2002
(Show Context)
Citation Context ... squared product ([0, 1] × X) 2 . There are standard ways of constructing such Mercer kernels from Mercer kernels on [0, 1] 2 and X 2 (see, e.g., the description of tensor products and direct sums in =-=[13, 11]-=-). For Sn to be continuous, we have to require that K be forecast-continuous in the following sense: for all x ∈ X and all (p ′ , x ′ ) ∈ [0, 1] × X, K((p, x), (p ′ , x ′ )) is continuous as a functio... |

547 |
Stochastic Processes
- Doob
- 1953
(Show Context)
Citation Context ...n satisfies the following two properties: Validity Suppose Skeptic’s strategy is measurable and pn are obtained from P; Kn then form a nonnegative martingale w.r. to P. According to Doob’s inequality =-=[14, 3]-=-, for any positive constant C, sup n Kn ≥ C with Pprobability at most 1/C. (If Forecaster is doing a bad job according to the testing interpretation, he is also doing a bad job from the standard point... |

136 | Probability and Finance: It’s Only a Game
- Shafer, Vovk
- 2001
(Show Context)
Citation Context ...t that 1 lim n→∞ n n� (yi − pi) = 0. (1) i=1 Such results will be called laws of probability and the existing body of laws of probability will be called classical probability theory. In §2, following =-=[12]-=-, we formalize Forecaster’s goal by adding a third player, Skeptic, who is allowed to gamble at the odds given by Forecaster’s probabilities. We state a result from [14] and [12] suggesting that Skept... |

106 |
Etude Critique de la Notion de Collectif
- Ville
- 1939
(Show Context)
Citation Context ...ility theory. In §2, following [12], we formalize Forecaster’s goal by adding a third player, Skeptic, who is allowed to gamble at the odds given by Forecaster’s probabilities. We state a result from =-=[14]-=- and [12] suggesting that Skeptic’s gambling strategies can be used as tests of agreement between pn and yn and that all tests of agreement between pn and yn can be expressed as Skeptic’s gambling str... |

68 | Asymptotic calibration
- Foster, Vohra
- 1998
(Show Context)
Citation Context ...r Skeptic is as fundamental in defensive forecasting as the possibility of mixing strategies for Forecaster in prediction with expert advice. This paper continues the work started by Foster and Vohra =-=[4]-=- and later developed in, e.g., [6, 10, 18] (the last paper replaces the von Mises–style framework of the previous papers with a martingale framework, as in this paper). The approach of this paper is s... |

63 | Competitive on-line statistics
- Vovk
(Show Context)
Citation Context ...ing has been developing without any explicit connections with classical probability theory. Defensive forecasting is indirectly related, in a sense dual, to prediction with expert advice (reviewed in =-=[15]-=-, §4) and its special case, Bayesian prediction. In prediction with expert advice one starts with a given loss function and tries to make predictions that lead to a small loss as measured by that loss... |

43 |
Probability forecasting
- Dawid
- 1986
(Show Context)
Citation Context ... condition for good forecasts: for example, a forecaster who ignores the objects xn can be perfectly calibrated, no matter how much useful information xn contain. (Cf. the discussion of resolution in =-=[2]-=-; we prefer not to use the term “resolution”, which is too closely connected with the very special way of probability forecasting based on sorting and labeling.) It is easy to make the algorithm of th... |

39 | Deterministic calibration and Nash equilibrium
- Kakade, Foster
- 2004
(Show Context)
Citation Context ... To achieve unbiasedness in the small in this stronger sense, randomization appears necessary (see, e.g., [18]). It is interesting that already a little bit of randomization suffices, as explained in =-=[5]-=-. 5 Simplified algorithm Let us assume first that objects are absent, |X| = 1. It was observed empirically that the performance of defensive forecasting strategies with a fixed ǫ does not depend on ǫ ... |

29 |
Calibration with many checking rules
- Sandroni, Smorodinsky, et al.
- 2003
(Show Context)
Citation Context ...ensive forecasting as the possibility of mixing strategies for Forecaster in prediction with expert advice. This paper continues the work started by Foster and Vohra [4] and later developed in, e.g., =-=[6, 10, 18]-=- (the last paper replaces the von Mises–style framework of the previous papers with a martingale framework, as in this paper). The approach of this paper is similar to that of the recent paper [5], wh... |

22 | 2004), “Good Randomized Sequential Probability Forecasting is Always Possible,” mimeo
- Vovk, Shafer
(Show Context)
Citation Context ...hieve if I are allowed to be indicator functions of intervals (such as [0, 0.5) and [0.5, 1]). To achieve unbiasedness in the small in this stronger sense, randomization appears necessary (see, e.g., =-=[18]-=-). It is interesting that already a little bit of randomization suffices, as explained in [5]. 5 Simplified algorithm Let us assume first that objects are absent, |X| = 1. It was observed empirically ... |

16 | Any inspection is manipulable
- Lehrer
- 2001
(Show Context)
Citation Context ...ensive forecasting as the possibility of mixing strategies for Forecaster in prediction with expert advice. This paper continues the work started by Foster and Vohra [4] and later developed in, e.g., =-=[6, 10, 18]-=- (the last paper replaces the von Mises–style framework of the previous papers with a martingale framework, as in this paper). The approach of this paper is similar to that of the recent paper [5], wh... |

11 | Self-calibrating priors do not exist (with discussion - Oakes - 1985 |

8 |
Self-calibrating priors do not exist: Comment
- Dawid
- 1985
(Show Context)
Citation Context ...d empirically. 1 Introduction Probability forecasting can be thought of as a game between two players, Forecaster and Reality: FOR n = 1, 2, . . .: Reality announces xn ∈ X. Forecaster announces pn ∈ =-=[0, 1]-=-. Reality announces yn ∈ {0, 1}. On each round, Forecaster predicts Reality’s move yn chosen from the label space, always taken to be {0, 1} in this paper. His move, the probability forecast pn, can b... |

7 |
Defensive forecasting, The Game-Theoretic Probability and Finance project, http://proba bilityandfinance.com, Working Paper #8, September 2004 (revised January 2005). A shorter version is published
- Vovk, Takemura, et al.
- 2004
(Show Context)
Citation Context ...n(Sn))/2. Read yn ∈ {0, 1}. 12sComputer experiments reported in [16] show that the K29 algorithm performs well on a standard benchmark data set. For a theoretical discussion of the K29 algorithm, see =-=[19]-=- (Appendix) and [17]. 6 Related work and directions of further research This paper’s methods connect two areas that have been developing independently so far: probability forecasting and classical pro... |

6 |
Verification of probabilistic predictions: A brief review
- MURPHY, EPSTEIN
- 1967
(Show Context)
Citation Context ...this section we will draw on the excellent survey [2]. We will see how Forecaster defeats increasingly sophisticated strategies for Skeptic. 4.1 Unbiasedness in the large Following Murphy and Epstein =-=[7]-=-, we say that Forecaster is unbiased in the large if (1) holds. Let us first consider the one-sided relaxed version of this property limsup n→∞ 1 n i=1 n� (yi − pi) ≤ ǫ. (4) i=1 The strategy for Skept... |

5 |
Non-asymptotic calibration and resolution, The GameTheoretic Probability and Finance project, http://probabilityandfi nance.com, Working Paper #13
- Vovk
- 2005
(Show Context)
Citation Context ...{0, 1}. 12sComputer experiments reported in [16] show that the K29 algorithm performs well on a standard benchmark data set. For a theoretical discussion of the K29 algorithm, see [19] (Appendix) and =-=[17]-=-. 6 Related work and directions of further research This paper’s methods connect two areas that have been developing independently so far: probability forecasting and classical probability theory. It ... |

2 |
Defensive forecasting for a benchmark data set, The Game-Theoretic Probability and Finance project, http://probabilityandfi nance.com, Working Paper #9
- Vovk
- 2004
(Show Context)
Citation Context ...2 FOR n = 1, 2, . . .: Read xn ∈ X. Define Sn(p) as per (9). Output any root p of Sn(p) = 0 as pn; if there are no roots, pn := (1 + sign(Sn))/2. Read yn ∈ {0, 1}. 12sComputer experiments reported in =-=[16]-=- show that the K29 algorithm performs well on a standard benchmark data set. For a theoretical discussion of the K29 algorithm, see [19] (Appendix) and [17]. 6 Related work and directions of further r... |