Abstract

In this paper, we study the problem of learning in the presence of classification noise in the probabilistic learning model of Valiant and its variants. In order to identify the class of “robust” learning algorithms in the most general way, we formalize a new but related model of learning from statistical queries. Intuitively, in this model, a learning algorithm is forbidden to examine individual examples of the unknown target function, but is given access to an oracle providing estimates of probabilities over the sample space of random examples. One of our main results shows that any class of functions learnable from statistical queries is in fact learnable with classification noise in Valiant’s model, with a noise rate approaching the information-theoretic barrier of 1/2. We then demonstrate the generality of the statistical query model, showing that practically every class learnable in Valiant’s model and its variants can also be learned in the new model (and thus can be learned in the presence of noise). A notable exception to this statement is the class of parity functions, which we prove is not learnable from statistical queries, and for which no noise-tolerant algorithm is known.

Citations

1515	A Theory of the Learnable - Valiant - 1984
816	On the uniform convergence of relative frequencies of events to their probabilities. Theory of Prob. and its - Vapnik, Chervonenkis - 1971
565	Learnability and the Vapnik Chervonenkis dimension - Blumer, Ehrenfeucht, et al. - 1989
507	An Introduction to Computational Learning Theory - Kearns, Vazirani - 1994
436	Perceptrons; An introduction to computational geometry - Minsky, Papert - 1969
347	Learning decision lists - Rivest - 1987
253	Constant Depth Circuits, Fourier Transform, and Learnability - Linial, Mansour, et al. - 1993
207	Quantifying inductive bias: AI learning algorithms and Valiant’s learning framework - Haussler - 1988
197	Learning from noisy examples - Angluin, Laird - 1988
182	Efficient distributionfree learning of probabilistic concepts - Kearns, Schapire - 1994
182	Computational limitations on learning from examples - Pitt, Valiant - 1988
180	A General Lower Bound on the Number of Examples Needed - Ehrenfeucht, Haussler, et al. - 1989
158	Learning in the presence of malicious errors - Kearns, Li - 1993
155	On the Learnability of Boolean Formula - Kearns, Li, et al. - 1987
114	Learning disjunctions of conjunctions - Valiant - 1985
105	Weakly learning dnf and characterizing statistical query learning using fourier analysis - Blum, Furst, et al. - 1994
59	Learning integer lattices - Helmbold, Sloan, et al. - 1992
57	The Design and Analysis of Efficient Learning Algorithms - Schapire - 1991
56	The Computational Complexity of Machine Learning - Kearns - 1990
51	A Polynomial-time Algorithm for Learn-ing Noisy Linear Threshold Functions - Blum, Frieze, et al. - 1996
48	Types of noise in data for concept learning - Sloan - 1988
46	Learning from Good and Bad Data - Laird - 1988
41	Statistical mechanics of learning from examples - Seung, Sompolinsky, et al. - 1992
41	General bounds on statistical query learning and PAC learning with noise via hypothesis boosting - Aslam, Decatur - 1998
33	Learning noisy perceptrons by a perceptron in polynomial time - Cohen - 1997
30	Specification and simulation of statistical query algorithms for efficiency and noise tolerance - Aslam, Decatur - 1995
25	On learning ring-sumexpansions - Fischer, Simon - 1992
23	Improved learning of ac 0 functions - Furst, Jackson, et al. - 1991
20	A polynomial-time algorithm for learning k--variable pattern languages from examples - Kearns, Pitt - 1991
20	Learning from Good and Bad Data. Kluwer international series in engineering and computer science - Laird - 1988
17	Improved learning of AC ° functions - Furst, Jackson, et al. - 1991
13	Learning monotone k-� DNF formulas on product distributions - HANCOCK, MANSOUR - 1991
12	Learning probabilistic read-once formulas on product distributions - Schapire - 1994
11	Three unfinished works on the optimal storage capacity of networks. 3 - Gardner, Derrida - 1989
9	Algorithmic Learning of Formal Languages and Decision Trees - Sakakibara - 1991
8	The transition to perfect generalization in perceptrons - Baum, Lyuu - 1991
2	and Yuh-Dauh Lyuu. The transition to perfect generalization in perceptrons - Baum - 1991