Abstract

Abstract. A Bayesian model of learning to learn by sampling from multiple tasks is presented. The multiple tasks are themselves generated by sampling from a distribution over an environment of related tasks. Such an environment is shown to be naturally modelled within a Bayesian context by the concept of an objective prior distribution. It is argued that for many common machine learning problems, although in general we do not know the true (objective) prior for the problem, we do have some idea of a set of possible priors to which the true prior belongs. It is shown that under these circumstances a learner can use Bayesian inference to learn the true prior by learning sufficiently many tasks from the environment. In addition, bounds are given on the amount of information required to learn a task when it is simultaneously learnt with several other tasks. The bounds show that if the learner has little knowledge of the true prior, but the dimensionality of the true prior is small, then sampling multiple tasks is highly advantageous. The theory is applied to the problem of learning a common feature set or equivalently a low-dimensional-representation (LDR) for an environment of related tasks.

Citations

6514	Elements of Information Theory - Cover, Thomas - 1991
917	Statistical decision theory and Bayesian analysis - Berger - 1985
194	Probalistic interpretation of feedforward classification network outputs, with relationships to statistical pattern recognition - Bridle - 1990
158	Approximation capabilities of multilayer feedforward networks, Neural Networks 4 - Hornik - 1991
134	The evidence framework applied to classification networks - MacKay - 1992
92	Information–theoretic asymptotics of Bayes methods - Clarke, Barron - 1990
82	Learning from Hints in Neural Networks - Abu–Mostafa - 1990
81	Learning internal representations - Baxter - 1995
57	Learning one more thing - Thrun, Mitchell - 1995
52	Fat-shattering and the learnability of real-valued functions - Bartlett, Long, et al. - 1996
52	Learning many related tasks at the same time with backpropagation - Caruana - 1995
25	Function learning from interpolation - Anthony, Bartlett
23	Some History of the Hierarchical Bayesian Methodology," in Good Thinking: The Foundations of Probability and its Applications - GOOD - 1983
14	Learning model bias - Baxter - 1996
14	General bounds on the mutual information between a parameter and n conditionally independent observations - Haussler, Opper - 1995
11	A Bayesian/information theoretic model of bias learning - Baxter - 1996
10	The need for biases in learning generalisations - Mitchell - 1980
6	Jeffreys' Prior is Asymptotically Least Favourable under Entropy Risk - Clarke, Barron - 1994
4	A model for bias learning - Baxter - 1997
2	Multivariate Estimation - Berger - 1986
1	Reconstructing a neural network from its output - Fefferman - 1994