Results 1  10
of
19
Asymptotic calibration
, 1998
"... Can we forecast the probability of an arbitrary sequence of events happening so that the stated probability of an event happening is close to its empirical probability? We can view this prediction problem as a game played against Nature, where at the beginning of the game Nature picks a data sequenc ..."
Abstract

Cited by 71 (4 self)
 Add to MetaCart
Can we forecast the probability of an arbitrary sequence of events happening so that the stated probability of an event happening is close to its empirical probability? We can view this prediction problem as a game played against Nature, where at the beginning of the game Nature picks a data sequence and the forecaster picks a forecasting algorithm. If the forecaster is not allowed to randomise, then Nature wins; there will always be data for which the forecaster does poorly. This paper shows that, if the forecaster can randomise, the forecaster wins in the sense that the forecasted probabilities and the empirical probabilities can be made arbitrarily close to each other.
Probabilistic forecasts, calibration and sharpness
 Journal of the Royal Statistical Society Series B
, 2007
"... Summary. Probabilistic forecasts of continuous variables take the form of predictive densities or predictive cumulative distribution functions. We propose a diagnostic approach to the evaluation of predictive performance that is based on the paradigm of maximizing the sharpness of the predictive dis ..."
Abstract

Cited by 41 (15 self)
 Add to MetaCart
Summary. Probabilistic forecasts of continuous variables take the form of predictive densities or predictive cumulative distribution functions. We propose a diagnostic approach to the evaluation of predictive performance that is based on the paradigm of maximizing the sharpness of the predictive distributions subject to calibration. Calibration refers to the statistical consistency between the distributional forecasts and the observations and is a joint property of the predictions and the events that materialize. Sharpness refers to the concentration of the predictive distributions and is a property of the forecasts only. A simple theoretical framework allows us to distinguish between probabilistic calibration, exceedance calibration and marginal calibration. We propose and study tools for checking calibration and sharpness, among them the probability integral transform histogram, marginal calibration plots, the sharpness diagram and proper scoring rules. The diagnostic approach is illustrated by an assessment and ranking of probabilistic forecasts of wind speed at the Stateline wind energy centre in the US Pacific Northwest. In combination with crossvalidation or in the time series context, our proposal provides very general, nonparametric alternatives to the use of information criteria for model diagnostics and model selection.
Deterministic calibration and Nash equilibrium
 Proceedings of the Seventeenth Annual Conference on Learning Theory, volume 3120 of Lecture Notes in Computer Science
, 2004
"... Abstract. We provide a natural learning process in which the joint frequency of empirical play converges into the set of convex combinations of Nash equilibria. In this process, all players rationally choose their actions using a public prediction made by a deterministic, weakly calibrated algorithm ..."
Abstract

Cited by 39 (2 self)
 Add to MetaCart
Abstract. We provide a natural learning process in which the joint frequency of empirical play converges into the set of convex combinations of Nash equilibria. In this process, all players rationally choose their actions using a public prediction made by a deterministic, weakly calibrated algorithm. Furthermore, the public predictions used in any given round of play are frequently close to some Nash equilibrium of the game. 1
Conditional Universal Consistency
, 1997
"... Each period, a player must choose an action without knowing the outcome that will be chosen by "Nature," according to an unknown and possibly historydependent stochastic rule. We discuss have a class of procedures that assign observations to categories, and prescribe a simple randomized v ..."
Abstract

Cited by 33 (0 self)
 Add to MetaCart
Each period, a player must choose an action without knowing the outcome that will be chosen by "Nature," according to an unknown and possibly historydependent stochastic rule. We discuss have a class of procedures that assign observations to categories, and prescribe a simple randomized variation of fictitious play within each category. These procedures are "conditionally consistent," in the sense of yielding almost as high a timeaverage payoff as could be obtained if the player chose knowing the conditional distributions of actions given categories. Moreover given any alternative procedure, there is a conditionally consistent procedure whose performance is no more than epsilon worse regardless of the discount factor. Cycles can persist if all players classify histories in the same way; however in an example, where players classify histories differently, the system converges to a Nash equilibrium. We also argue that in the long run the timeaverage of play should resemble a correlated equilibrium.
An Easier Way to Calibrate
, 1995
"... This document is copyrighted by the authors. You may freely reproduce and distribute it electronically or in print, provided it is distributed in its entirety, including this copyright notice. 1 The authors are grateful for financial support from NSF grants SBR9223320, SBR9223175, SBR9409180 and ..."
Abstract

Cited by 20 (0 self)
 Add to MetaCart
This document is copyrighted by the authors. You may freely reproduce and distribute it electronically or in print, provided it is distributed in its entirety, including this copyright notice. 1 The authors are grateful for financial support from NSF grants SBR9223320, SBR9223175, SBR9409180 and the UCLA Academic Senate. This paper benefited from conversations with Glen Ellison and Dean Foster
A Geometric Proof of Calibration
, 2009
"... We provide yet another proof of the existence of calibrated forecasters; it has two merits. First, it is valid for an arbitrary finite number of outcomes. Second, it is short and simple and it follows from a direct application of Blackwell’s approachability theorem to carefully chosen vectorvalued ..."
Abstract

Cited by 9 (6 self)
 Add to MetaCart
We provide yet another proof of the existence of calibrated forecasters; it has two merits. First, it is valid for an arbitrary finite number of outcomes. Second, it is short and simple and it follows from a direct application of Blackwell’s approachability theorem to carefully chosen vectorvalued payoff function and convex target set. Our proof captures the essence of existing proofs based on approachability (e.g., the proof by Foster [1999] in case of binary outcomes) and highlights the intrinsic connection between approachability and calibration.
Online calibrated forecasts: Memory efficiency versus universality for learning in games
 MACH LEARN
, 2006
"... We provide a simple learning process that enables an agent to forecast a sequence of outcomes. Our forecasting scheme, termed tracking forecast, is based on tracking the past observations while emphasizing recent outcomes. As opposed to other forecasting schemes, we sacrifice universality in favor ..."
Abstract

Cited by 8 (8 self)
 Add to MetaCart
We provide a simple learning process that enables an agent to forecast a sequence of outcomes. Our forecasting scheme, termed tracking forecast, is based on tracking the past observations while emphasizing recent outcomes. As opposed to other forecasting schemes, we sacrifice universality in favor of a significantly reduced memory requirements. We show that if the sequence of outcomes has certain properties—it has some internal (hidden) state that does not change too rapidly—then the tracking forecast is weakly calibrated so that the forecast appears to be correct most of the time. For binary outcomes, this result holds without any internal state assumptions. We consider learning in a repeated strategic game where each player attempts to compute some forecast of the opponent actions and play a best response to it. We show that if one of the players uses a tracking forecast, while the other player uses a standard learning algorithm (such as exponential regret matching or smooth fictitious play), then the player using the tracking forecast obtains the best response to the actual play of the other players. We further show that if both players use tracking forecast, then under certain conditions on the game matrix, convergence to a Nash
The complexity of forecast testing
 Electronic Colloquium on Computational Complexity
, 2006
"... The copyright to this Article is held by the Econometric Society. It may be downloaded, printed and reproduced only for educational or research purposes, including use in course packs. No downloading or copying may be done for any commercial purpose without the explicit permission of the Econometric ..."
Abstract

Cited by 7 (0 self)
 Add to MetaCart
The copyright to this Article is held by the Econometric Society. It may be downloaded, printed and reproduced only for educational or research purposes, including use in course packs. No downloading or copying may be done for any commercial purpose without the explicit permission of the Econometric Society. For such commercial purposes contact the Office of the Econometric Society (contact information may be found at the website http://www.econometricsociety.org or in the back cover of Econometrica). This statement must the included on all copies of this Article that are made available electronically or in any other format. Econometrica, Vol. 77, No. 1 (January, 2009), 93–105
Switching supervisory control using calibrated forecasts,” 2007, submitted for publication
"... Abstract—In this paper, we approach supervisory control as an online decision problem. In particular, we introduce “calibrated forecasts ” as a mechanism for controller selection in supervisory control. The forecasted quantity is a candidate controller’s performance level, or reward, over finite imp ..."
Abstract

Cited by 6 (4 self)
 Add to MetaCart
Abstract—In this paper, we approach supervisory control as an online decision problem. In particular, we introduce “calibrated forecasts ” as a mechanism for controller selection in supervisory control. The forecasted quantity is a candidate controller’s performance level, or reward, over finite implementation horizon. Controller selection is based on using the controller with the maximum calibrated forecast of the reward. The proposed supervisor does not perform a prerouted search of candidate controllers and does not require the presence of exogenous inputs for excitation or identification. Assuming the existence of a stabilizing controller within the set of candidate controllers, we show that under the proposed supervisory controller, the output of the system remains bounded for any bounded disturbance, even if the disturbance is chosen in an adversarial manner. The use of calibrated forecasts enables one to establish overall performance guarantees for the supervisory scheme even though nonstabilizing controllers may be persistently selected by the supervisor because of the effects of initial conditions, exogenous disturbances, or random selection. The main results are obtained for a general class of system dynamics and specialized to linear systems. Index Terms—Adaptive control, calibrated forecast, machine learning, supervisory control. I.
Is there an Elegant Universal Theory of Prediction?
 IDSIA / USISUPSI DALLE MOLLE INSTITUTE FOR ARTIFICIAL INTELLIGENCE. GALLERIA 2, 6928
, 2006
"... Solomonoff’s inductive learning model is a powerful, universal and highly elegant theory of sequence prediction. Its critical flaw is that it is incomputable and thus cannot be used in practice. It is sometimes suggested that it may still be useful to help guide the development of very general and p ..."
Abstract

Cited by 4 (0 self)
 Add to MetaCart
Solomonoff’s inductive learning model is a powerful, universal and highly elegant theory of sequence prediction. Its critical flaw is that it is incomputable and thus cannot be used in practice. It is sometimes suggested that it may still be useful to help guide the development of very general and powerful theories of prediction which are computable. In this paper it is shown that although powerful algorithms exist, they are necessarily highly complex. This alone makes their theoretical analysis problematic, however it is further shown that beyond a moderate level of complexity the analysis runs into the deeper problem of Gödel incompleteness. This limits the power of mathematics to analyse and study prediction algorithms, and indeed intelligent systems in general.