## Complexity Approximation Principle (1999)

Venue: | Computer Journal |

Citations: | 4 - 3 self |

### BibTeX

@ARTICLE{Vovk99complexityapproximation,

author = {Vladimir Vovk and Alexander Gammerman},

title = {Complexity Approximation Principle},

journal = {Computer Journal},

year = {1999},

volume = {42},

pages = {318--322}

}

### OpenURL

### Abstract

INTRODUCTION The subject of this note is another inductive principle, which can be regarded as a direct generalization of the minimum description length (MDL) and minimum message length (MML) principles. We will describe the work started at the Computer Learning Research Centre (Royal Holloway, University of London) related to this new principle, which we call the complexity approximation principle (CAP). Both MDL and MML principles can be interpreted as Kolmogorov complexity approximation principles (as explained in Rissanen [1, 2] and Wallace and Freeman [3]; see also [4]). It is shown in [5] and [6] that it is possible to generalize Kolmogorov complexity to describe the optimal performance in different `games of prediction'. Using this general notion, called predictive complexity,itis straightforward to extend the MDL and MML principles to our more general CAP. In Section 2 we define predictive complexity, in Section 3 several examples are given and in Section 4

### Citations

10096 | Statistical Learning Theory
- Vapnik
- 1998
(Show Context)
Citation Context ... Suppose we are given some set of feasible prediction strategies (for example, in the situation of Example 2, our set might contain the polynomials of all possible degrees, or support vector machines =-=[17]-=- with different kernels) and our goal is to choose the `best' strategy in this set in view of some data sequence. The following is a reasonable heuristic for avoiding over- (and under-) fitting: COMPL... |

1786 | An Introduction to Kolmogorov Complexity and Its Applications (Second Edition
- Li, Vitanyi
- 1997
(Show Context)
Citation Context ...ssumed to be computable. Within our general framework of games of prediction, Kolmogorov complexity (more accurately, its predictive variant, known as the minus log of Levin's apriorisemimeasure; see =-=[8, 9, 10]-=-) describes complexity with respect to a particular game, the so-called `log-loss' game. There are, however, many other interesting games; e.g. Example 1 involves the so-called square-loss game. A `da... |

561 |
Three approaches to the quantitative definition of information. Prob
- Kolmogorov
- 1965
(Show Context)
Citation Context ...PLEXITY A typical question answered by the theory of Kolmogorov complexity is: what is the smallest size (in bits) of a file from which one can extract some text, say `War and Peace'? (See Kolmogorov =-=[7]-=-. At first this question might appear illposed: the shortest description of any particular text can be made zero, since the whole text can be built into the decoder; we are assuming that the reader kn... |

439 |
A universal prior for integers and estimation by minimum description length
- Rissanen
- 1983
(Show Context)
Citation Context ...s new principle, which we call the complexity approximation principle (CAP). Both MDL and MML principles can be interpreted as Kolmogorov complexity approximation principles (as explained in Rissanen =-=[1, 2]-=- and Wallace and Freeman [3]; see also [4]). It is shown in [5] and [6] that it is possible to generalize Kolmogorov complexity to describe the optimal performance in different `games of prediction'. ... |

269 |
Aggregating strategies
- Vovk
- 1990
(Show Context)
Citation Context ...ixable for some #>0. It is known that many popular games, such as the log-loss game, square-loss game, Cover's game, long-short game, Kullback--Leibler game, # 2 game, Hellinger game etc. (see, e.g., =-=[6, 11, 12, 13]-=-; some of these games will be described below), are perfectly mixable. LEMMA 1. There exists a universal measure of predictive complexity for any perfectly mixable game. (Remember that we always assum... |

112 | A game of prediction with expert advice
- Vovk
- 1995
(Show Context)
Citation Context ...ixable for some #>0. It is known that many popular games, such as the log-loss game, square-loss game, Cover's game, long-short game, Kullback--Leibler game, # 2 game, Hellinger game etc. (see, e.g., =-=[6, 11, 12, 13]-=-; some of these games will be described below), are perfectly mixable. LEMMA 1. There exists a universal measure of predictive complexity for any perfectly mixable game. (Remember that we always assum... |

95 | Universal portfolios with side information
- Cover, Ordentlich
- 1996
(Show Context)
Citation Context ...artificially restricting the range of possible values of y t .) A further interesting class of perfectly mixable games is provided by financial markets, such as Cover's game (see Cover and Ordentlich =-=[16]-=-, Vovk and Watkins [6]), the longshort game [6], the variants of these games with transaction costs, etc. It is interesting that KM is a special case of the predictive complexity in Cover's game (see ... |

61 | Hypothesis selection and testing by the MDL principle
- Rissanen
- 1999
(Show Context)
Citation Context ...be defined `to within an additive constant' any more; in the case of the absolute-loss game we will be able to get away with `to within a logarithm') and fruitfully apply CAP. Analogously to Rissanen =-=[18]-=-, we can generalize CAP to choosing a pool of strategies rather than one strategy. An important task now is to compare, theoretically and experimentally, CAP with other model selection principles (see... |

51 | Tight worst-case loss bounds for predicting with expert advice
- Haussler, Kivinen, et al.
- 1995
(Show Context)
Citation Context ...ixable for some #>0. It is known that many popular games, such as the log-loss game, square-loss game, Cover's game, long-short game, Kullback--Leibler game, # 2 game, Hellinger game etc. (see, e.g., =-=[6, 11, 12, 13]-=-; some of these games will be described below), are perfectly mixable. LEMMA 1. There exists a universal measure of predictive complexity for any perfectly mixable game. (Remember that we always assum... |

41 | Stochastic complexity (with discussion - Rissanen - 1987 |

26 |
Universal portfolio selection
- Vovk, Watkins
- 1998
(Show Context)
Citation Context ...). Both MDL and MML principles can be interpreted as Kolmogorov complexity approximation principles (as explained in Rissanen [1, 2] and Wallace and Freeman [3]; see also [4]). It is shown in [5] and =-=[6]-=- that it is possible to generalize Kolmogorov complexity to describe the optimal performance in different `games of prediction'. Using this general notion, called predictive complexity,itis straightfo... |

26 |
Measures of complexity of finite objects (axiomatic description
- Levin
- 1976
(Show Context)
Citation Context ...ty K G .Foreveryz # (X Y ) # , K G (z) will be called the `predictive complexity' of z in G. REMARK 1. Despite the fact that Kolmogorovcomplexity is just one example of predictive complexity, Levin's =-=[15]-=- results make it plausible that any predictive complexity can be reduced to Kolmogorov complexity (more accurately, its variant KM). It remains unclear, however, whether this reduction is of any pract... |

16 | Algorithmic complexity and stochastic properties of finite binary sequences
- V’yugin
(Show Context)
Citation Context ...ssumed to be computable. Within our general framework of games of prediction, Kolmogorov complexity (more accurately, its predictive variant, known as the minus log of Levin's apriorisemimeasure; see =-=[8, 9, 10]-=-) describes complexity with respect to a particular game, the so-called `log-loss' game. There are, however, many other interesting games; e.g. Example 1 involves the so-called square-loss game. A `da... |

15 |
Estimation and inference by compact coding (with discussion
- Wallace, Freeman
- 1987
(Show Context)
Citation Context ...the complexity approximation principle (CAP). Both MDL and MML principles can be interpreted as Kolmogorov complexity approximation principles (as explained in Rissanen [1, 2] and Wallace and Freeman =-=[3]-=-; see also [4]). It is shown in [5] and [6] that it is possible to generalize Kolmogorov complexity to describe the optimal performance in different `games of prediction'. Using this general notion, c... |

12 | Vovk V., Kolmogorov Complexity: Sources, Theory and Applications
- Gammerman
- 1999
(Show Context)
Citation Context ... approximation principle (CAP). Both MDL and MML principles can be interpreted as Kolmogorov complexity approximation principles (as explained in Rissanen [1, 2] and Wallace and Freeman [3]; see also =-=[4]-=-). It is shown in [5] and [6] that it is possible to generalize Kolmogorov complexity to describe the optimal performance in different `games of prediction'. Using this general notion, called predicti... |

9 |
Algorithmic entropy (complexity) of finite objects and its application to defining randomness and amount of information, Semiotika i
- V’yugin
- 1981
(Show Context)
Citation Context ...ssumed to be computable. Within our general framework of games of prediction, Kolmogorov complexity (more accurately, its predictive variant, known as the minus log of Levin's apriorisemimeasure; see =-=[8, 9, 10]-=-) describes complexity with respect to a particular game, the so-called `log-loss' game. There are, however, many other interesting games; e.g. Example 1 involves the so-called square-loss game. A `da... |

4 |
Discuss
- Clarke, Terentjev
- 1999
(Show Context)
Citation Context ... we can generalize CAP to choosing a pool of strategies rather than one strategy. An important task now is to compare, theoretically and experimentally, CAP with other model selection principles (see =-=[19]-=-). CAP is an active field of research at the Computer Learning Research Centre and the reader will be able to find information about new results at its web site http://clrc.rhbnc.ac.uk. ACKNOWLEDGEMEN... |

2 |
Probability theory for the Brier game. Accepted for publication in Theoretical Computer Science. Preliminary version in Ming Li and Akira
- Vovk
- 1997
(Show Context)
Citation Context ...ple (CAP). Both MDL and MML principles can be interpreted as Kolmogorov complexity approximation principles (as explained in Rissanen [1, 2] and Wallace and Freeman [3]; see also [4]). It is shown in =-=[5]-=- and [6] that it is possible to generalize Kolmogorov complexity to describe the optimal performance in different `games of prediction'. Using this general notion, called predictive complexity,itis st... |

1 |
The complexity of finite objects and the development of the THE
- Zvonkin, Levin
- 1970
(Show Context)
Citation Context ...ly.) The idea of the proof of Lemma 1 is to apply the `aggregating algorithm' [11, 13] to some effective enumeration of all measures of predictive complexity; for details see [6] (Section 7.6). Levin =-=[14]-=- proved the existence of a universal measure of predictive complexity for the log-loss game (in which the notion of a universal measure of predictive complexity is equivalent to the minus logarithm of... |