Abstract

We study annealed theories of learning boolean functions using a concept class of finite cardinality. The naive annealed theory can be used to derive a universal learning curve bound for zero temperature learning, similar to the inverse square root bound from the Vapnik-Chervonenkis theory. Tighter, nonuniversal learning curve bounds are also derived. A more refined annealed theory leads to still tighter bounds, which in some cases are very similar to results previously obtained using one-step replica symmetry breaking. 1. Introduction The annealed approximation 1 has proven to be an invaluable tool for studying the statistical mechanics of learning from examples. Previously it was found that the annealed approximation gave qualitatively correct results for several models of perceptrons learning realizable rules. 2 Because of its simplicity relative to the full quenched theory, the annealed approximation has since been used in studies of more complicated multilayer architectures. ...

Citations

687	Estimation of dependences based on empirical data - Vapnik - 1982
51	Rigorous learning curve bounds from statistical mechanics - Haussler, Kearns, et al. - 1994
41	Statistical mechanics of learning from examples - Seung, Sompolinsky, et al. - 1992
26	Statistical Theory of Learning a Rule - Györgyi, Tishby - 1990
23	Four types of learning curves - Amari, Fujita, et al. - 1992
20	Exhaustive Learning - Schwartz, Samalam, et al. - 1990
5	den Broeck. Systems that can learn from examples: Replica calculation of uniform convergence bounds for perceptrons. Phys - Engel, van - 1772
3	Generalization in a two-layer neural network - Kang, Oh, et al. - 1993
3	Generalization in fully connected committee machines - Schwarze, Hertz - 1993
2	Learning curves for error minimum and maximum likelihood algorithms - Kabashima, Shinomoto - 1992