Results 1  10
of
13
Bounds on the Sample Complexity of Bayesian Learning Using Information Theory and the VC Dimension
 Machine Learning
, 1994
"... In this paper we study a Bayesian or averagecase model of concept learning with a twofold goal: to provide more precise characterizations of learning curve (sample complexity) behavior that depend on properties of both the prior distribution over concepts and the sequence of instances seen by the l ..."
Abstract

Cited by 108 (12 self)
 Add to MetaCart
In this paper we study a Bayesian or averagecase model of concept learning with a twofold goal: to provide more precise characterizations of learning curve (sample complexity) behavior that depend on properties of both the prior distribution over concepts and the sequence of instances seen by the learner, and to smoothly unite in a common framework the popular statistical physics and VC dimension theories of learning curves. To achieve this, we undertake a systematic investigation and comparison of two fundamental quantities in learning and information theory: the probability of an incorrect prediction for an optimal learning algorithm, and the Shannon information gain. This study leads to a new understanding of the sample complexity of learning in several existing models. 1 Introduction Consider a simple concept learning model in which the learner attempts to infer an unknown target concept f , chosen from a known concept class F of f0; 1gvalued functions over an instance space X....
Compressive Sensing and Structured Random Matrices
 RADON SERIES COMP. APPL. MATH XX, 1–95 © DE GRUYTER 20YY
"... These notes give a mathematical introduction to compressive sensing focusing on recovery using ℓ1minimization and structured random matrices. An emphasis is put on techniques for proving probabilistic estimates for condition numbers of structured random matrices. Estimates of this type are key to ..."
Abstract

Cited by 59 (13 self)
 Add to MetaCart
These notes give a mathematical introduction to compressive sensing focusing on recovery using ℓ1minimization and structured random matrices. An emphasis is put on techniques for proving probabilistic estimates for condition numbers of structured random matrices. Estimates of this type are key to providing conditions that ensure exact or approximate recovery of sparse vectors using ℓ1minimization.
Uniform Ratio Limit Theorems for Empirical Processes
, 1995
"... . Analysis of nonparametric estimators is easier if one makes use of refined empirical process inequalities. These inequalities have not been widely appreciated or applied, perhaps because they lie buried quite deep in the empirical process literature. This paper seeks to explain a novel empirical p ..."
Abstract

Cited by 11 (0 self)
 Add to MetaCart
. Analysis of nonparametric estimators is easier if one makes use of refined empirical process inequalities. These inequalities have not been widely appreciated or applied, perhaps because they lie buried quite deep in the empirical process literature. This paper seeks to explain a novel empirical process technique that might make the results more accessible. A simple method is presented for proving maximal inequalities for empirical processes indexed by unbounded classes of functions. The main result is stated in the form of a tail bound for the ratio of a centered empirical measure empirical to a sum of various L 1 norms. The method avoids the usual chaining argument. The bound delivers the correct rates of convergence for the sorts of quantities encountered in nonparametric smoothing applications. Research partially supported by NSF Grant DMS9102286 Keywords and Phrases: Maximal inequalities for empirical process; ratio limit theorems; nonparametric bounds. Running head: Empirical...
DistributionDependent VapnikChervonenkis Bounds
, 1999
"... . VapnikChervonenkis (VC) bounds play an important role in statistical learning theory as they are the fundamental result which explains the generalization ability of learning machines. There have been consequent mathematical works on the improvement of VC rates of convergence of empirical mean ..."
Abstract

Cited by 6 (1 self)
 Add to MetaCart
. VapnikChervonenkis (VC) bounds play an important role in statistical learning theory as they are the fundamental result which explains the generalization ability of learning machines. There have been consequent mathematical works on the improvement of VC rates of convergence of empirical means to their expectations over the years. The result obtained by Talagrand in 1994 seems to provide more or less the final word to this issue as far as universal bounds are concerned. Though for fixed distributions, this bound can be practically outperformed. We show indeed that it is possible to replace the 2ffl 2 under the exponential of the deviation term by the corresponding Cram'er transform as shown by large deviations theorems. Then, we formulate rigorous distributionsensitive VC bounds and we also explain why these theoretical results on such bounds can lead to practical estimates of the effective VC dimension of learning structures. 1 Introduction and motivations One of t...
ON THE LIMITING DISTRIBUTIONS OF MULTIVARIATE DEPTHBASED RANK SUM STATISTICS AND RELATED TESTS
, 708
"... A depthbased rank sum statistic for multivariate data introduced ..."
Abstract

Cited by 1 (0 self)
 Add to MetaCart
A depthbased rank sum statistic for multivariate data introduced
Discussion of: Empirical Processes and Applications by Evarist Giné
, 1993
"... Introduction. First, let me congratulate Professor Gin'e on his lucid and enthusiastic lectures. He has done a wonderful job of conveying the feel and excitement connected with recent developments in empirical process theory and the application of this theory in statistics. His simple and elegant pr ..."
Abstract
 Add to MetaCart
Introduction. First, let me congratulate Professor Gin'e on his lucid and enthusiastic lectures. He has done a wonderful job of conveying the feel and excitement connected with recent developments in empirical process theory and the application of this theory in statistics. His simple and elegant presentation of the inequalities clearly shows their power for obtaining the basics of the theory. There has been tremendous progress over the past 15 years on empirical process theory  and in its applications to problems in statistics. As I tried to argue in my recent review article [Wellner (1992)] the time lag between the introduction of problems and their solution using modern empirical process techniques seems to be decreasing rapidly. This progress in empirical process theory has gone hand in hand with considerable progress in some of the related areas of probability theory. Three general areas in particular are: ffl Gaussian process theory ffl weak
Large Deviations Bounds for Empirical Processes
, 1999
"... VapnikChervonenkis bounds on speeds of convergence of empirical means to their expectations have been continuously improved over the years. The result obtained by M. Talagrand in 1994 [11] seems to provide the final word as far as universal bounds are concerned. However, for fixed families of under ..."
Abstract
 Add to MetaCart
VapnikChervonenkis bounds on speeds of convergence of empirical means to their expectations have been continuously improved over the years. The result obtained by M. Talagrand in 1994 [11] seems to provide the final word as far as universal bounds are concerned. However, for fixed families of underlying probability distributions, the exponential rate in the deviation term can be fairly improved by the more adequate Cramer transform, as shown by large deviations theorems.
Learning Using Information Theory and the VC Dimension
"... Abstract. In this paper we study a Bayesian or averagecase model of concept learning with a twofold goal: to provide more precise characterizations of learning curve (sample complexity) behavior that depend on properties of both the prior distribution over concepts and the sequence of instances see ..."
Abstract
 Add to MetaCart
Abstract. In this paper we study a Bayesian or averagecase model of concept learning with a twofold goal: to provide more precise characterizations of learning curve (sample complexity) behavior that depend on properties of both the prior distribution over concepts and the sequence of instances seen by the learner, and to smoothly unite in a common framework the popular statistical physics and VC dimension theories of learning curves. To achieve this, we undertake a systematic investigation and comparison of two fundamental quantities in learning and information theory: the probability of an incorrect prediction for an optimal learning algorithm, and the Shannon information gain. This study leads to a new understanding of the sample complexity of learning in several existing models.
Exact Rates In VapnikChervonenkis Bounds
"... VapnikChervonenkis bounds on speeds of uniform convergence of empirical means to their expectations have been continuously improved over the years since the precursory work in [26]. The result obtained by Talagrand in 1994 [21] seems to provide the final word as far as universal bounds are concerne ..."
Abstract
 Add to MetaCart
VapnikChervonenkis bounds on speeds of uniform convergence of empirical means to their expectations have been continuously improved over the years since the precursory work in [26]. The result obtained by Talagrand in 1994 [21] seems to provide the final word as far as universal bounds are concerned. However, in the case where there are some additional assumptions on the underlying probability distribution, the exponential rate of convergence can be fairly improved. Alexander [1] and Massart [15] have found better exponential rates (similar to those in BennettBernstein inequalities) under the assumption of a control on the variance of the empirical process. In this paper, the case of a particular distribution is considered for the empirical process indexed by a family of sets, and we provide the exact exponential rate based on large deviations theorems, as predicted by Azencott [2].
S02460203(02)000109/FLA EXACT RATES IN VAPNIK–CHERVONENKIS BOUNDS
, 2000
"... ABSTRACT. – Vapnik–Chervonenkis bounds on rates of uniform convergence of empirical means to their expectations have been continuously improved over the years since the precursory work in [26]. The result obtained by Talagrand in 1994 [21] seems to provide the final word as far as universal bounds a ..."
Abstract
 Add to MetaCart
ABSTRACT. – Vapnik–Chervonenkis bounds on rates of uniform convergence of empirical means to their expectations have been continuously improved over the years since the precursory work in [26]. The result obtained by Talagrand in 1994 [21] seems to provide the final word as far as universal bounds are concerned. However, in the case where there are some additional assumptions on the underlying probability distribution, the exponential rate of convergence can be fairly improved. Alexander [1] and Massart [15] have found better exponential rates (similar to those in Bennett–Bernstein inequalities) under the assumption of a control on the variance of the empirical process. In this paper, the case of a particular distribution is considered for the empirical process indexed by a family of sets, and we provide the exact exponential rate based on large deviations theorems, as predicted by Azencott [2]. © 2003 Éditions scientifiques et médicales Elsevier SAS MSC: 60E15; 60F10 RÉSUMÉ. – Les bornes de Vapnik–Chervonenkis sur les vitesses de convergence uniforme des moyennes empiriques vers leurs espérances ont fait l’objet de nombreuses améliorations depuis leur travail précurseur [26]. Le résultat obtenu par Talagrand en 1994 [21] semble