Abstract

A major problem in machine learning is that of inductive bias: how to choose a learner's hypothesis space so that it is large enough to contain a solution to the problem being learnt, yet small enough to ensure reliable generalization from reasonably-sized training sets. Typically such bias is supplied by hand through the skill and insights of experts. In this paper a model for automatically learning bias is investigated. The central assumption of the model is that the learner is embedded within an environment of related learning tasks. Within such an environment the learner can sample from multiple tasks, and hence it can search for a hypothesis space that contains good solutions to many of the problems in the environment. Under certain restrictions on the set of all hypothesis spaces available to the learner, we show that a hypothesis space that performs well on a sufficiently large number of training tasks will also perform well when learning novel tasks in the same environment. Exp...

Citations

1515	A Theory of the Learnable - Valiant - 1984
917	Statistical decision theory and Bayesian analysis - Berger - 1985
887	D: Bayesian Data Analysis - Gelman, Carlin, et al. - 1995
801	A probabilistic theory of pattern recognition - Devroye, Györfi, et al. - 1996
688	Estimation of dependencies based on empirical data - Vapnik - 1982
565	Learnability and the Vapnik Chervonenkis dimension - Blumer, Ehrenfeucht, et al. - 1989
457	Convergence of Stochastic Processes - Pollard - 1984
328	Multitask learning - Caruana - 1997
328	Decision theoretic generalization of the PAC model for neural net and other learning applications - Haussler - 1992
295	Real analysis and probability - Dudley - 2002
255	PL: Neural Network Learning: Theoretical Foundations - Anthony, Bartlett - 1999
252	Classical Descriptive Set Theory - Kechris - 1995
205	On the density of families of sets - Sauer - 1972
183	Probability Measures on Metric Spaces - Parthasarathy - 1967
156	The sample complexity of pattern classification with neural networks: the size of the weights is more important than the size of the network - Bartlett
151	Transfer of learning by composing solutions of elemental sequential tasks - Singh - 1992
113	Shift of bias for inductive concept learning - Utgoff - 1986
109	Is learning the n-th thing any easier than learning the first - Thrun - 1996
98	Finding structure in reinforcement learning - Thrun, Schwartz - 1995
81	Learning internal representations - Baxter - 1995
78	Learning to Learn - Thrun, Pratt - 1998
69	Discovering structure in multiple learning tasks: The TC algorithm - Thrun, O’Sullivan - 1996
68	A course on empirical processes - Dudley - 1984
62	Adapting bias by gradient descent: An incremental version of deltabar-delta - Sutton - 1992
57	Learning one more thing - Thrun, Mitchell - 1995
55	A bayesian/information theoretic model of learning to learn via multiple task sampling - Baxter - 1997
43	The Use of Knowledge in Analogy and Induction - Russell - 1989
38	Discriminability-Based Transfer between Neural Networks - Pratt - 1993
35	A method for learning from hints - Abu–Mostafa - 1993
25	Symbolic-neural systems and the use of hints for developing complex systems - Suddarth, Holden - 1991
23	Some History of the Hierarchical Bayesian Methodology," in Good Thinking: The Foundations of Probability and its Applications - GOOD - 1983
22	Rule-injection Hints as a Means of Improving Network Performance and Learning Time - Suddarth, Kergosien - 1990
19	Layered concept-learning and dynamically-variable bias management - Rendell, Seshu, et al. - 1987
17	The canonical distortion measure for vector quantization and function approximation - Baxter - 1998
17	The parallel transfer of task knowledge using dynamic learning rates based on a measure of relatedness, Connect - Silver, Mercer - 1996
15	Adaptive Generalisation and the Transfer of Knowledge,” University of Exeter: R257 - Sharkey, Sharkey - 1992
14	Solving a huge number of similar tasks: a combination of multi-task learning and hierarchical Bayesian modeling - Heskes - 1998
14	Distribution inequalities for the binomial law - Slud - 1977
13	Making a lowdimensional representation suitable for diverse tasks. Connection Science 8(2):205–224 - Intrator, Edelman - 1996
10	The need for biases in learning generalisations - Mitchell - 1980
6	Repeat learning using predicate invention - Khan, Muggleton, et al. - 1998
4	The canonical distortion measure in feature space and 1-NN classification - Baxter, Bartlett - 1998
2	The process of learning - Langford - 1989
1	Lower bounds on the VC-dimension of multi-layer threshold networks - Bartlett - 1993
1	Continual Learning in Reinforcement Environments. R. Oldenbourg Verlag - Ring - 1995
1	On a double inequality of the normal distribution - Tate - 1953