Abstract

Abstract. This paper addresses the problem of improving the accuracy of an hypothesis output by a learning algorithm in the distribution-free (PAC) learning model. A concept class is learnable (or strongly learnable) if, given access to a Source of examples of the unknown concept, the learner with high probability is able to output an hypothesis that is correct on all but an arbitrarily small fraction of the instances. The concept class is weakly learnable if the learner can produce an hypothesis that performs only slightly better than random guessing. In this paper, it is shown that these two notions of learnability are equivalent. A method is described for converting a weak learning algorithm into one that achieves arbitrarily high accuracy. This construction may have practical applications as a tool for efficiently converting a mediocre learning algorithm into one that performs extremely well. In addition, the construction has some interesting theoretical consequences, including a set of general upper bounds on the complexity of any strong learning algorithm as a function of the allowed error e.

Citations

1128	Probability inequalities for sums of bounded random variables - Hoeffding - 1963
582	Queries and concept learning - Angluin - 1988
565	Learnability and the Vapnik Chervonenkis dimension - Blumer, Ehrenfeucht, et al. - 1989
347	Learning decision lists - Rivest - 1987
279	Cryptographic limitations on learning boolean formulae and finite automata - Kearns, Valiant - 1994
231	Fast probabilistic algorithms for hamiltonian circuits and matchings - Angluin, Valiant - 1977
193	Finding patterns common to a set of strings - Angluin - 1980
182	Computational limitations on learning from examples - Pitt, Valiant - 1988
155	On the Learnability of Boolean Formula - Kearns, Li, et al. - 1987
89	Equivalence of models for polynomial learnability - HAUSSLER, KEARNS, et al. - 1991
58	Learning decision trees from random examples - Ehrenfeucht, Haussler - 1989
44	Predicting {0, 1}-functions on randomly drawn points - HAUSSLER, LITTLESTONE, et al. - 1994
41	On the necessity of Occam algorithms - Board, Pitt - 1992
33	Learning nested differences of intersection-closed concept classes - Helmbold, Sloan, et al. - 1989
25	Learning boolean formulae or finite automata is as hard as factoring - Kearns, Valiant - 1988
16	Space-bounded learning and the Vapnik-Chervonenkis dimension - Floyd - 1989
4	Some remarks about space-complexity of learning, and circuit complexity of recognizing - Boucheron, Sallantin - 1988
4	Expected mistake bounds for on-line learning algorithms. Unpublished manuscript - Haussler, Littlestone, et al. - 1987
3	The Computational Complexity of Machine Learning. Doctoral dissertation - Kearns - 1989
2	Predicting {0, l}-functions on randomly drawn points - Haussler, Littlestone, et al. - 1990
1	On learning a union of half spaces. Unpublished manuscript - Baum - 1989
1	Space efficient learning algorithms (Technical Report UCSC-CRL-88-2 - Haussler - 1988
1	Pattern languages are not learnable. Unpublished manuscript - Schapire - 1989
1	Occam's razor, Infotwtation Ptwcessing Letters - Blumer, Ehmnfeucht, et al. - 1987
1	Some remarks about space-complexity of learning, and circuit complexity of recognizing - Bouchemn, Sallantln - 1988
1	Learning decision trees from random exmnples, hfimnatiou and putation - Ehrenfeucht, Haussler - 1989
1	Probability inequalities for sums of bounded randent variables - Hoeffdiug - 1963
1	Space efficient learning algorithms (Technical Report UCSC-CRL-88-2 - Hanssler - 1988