A Bayesian/Information Theoretic Model of Bias Learning
This addresses the challenge of bias learning in multi-task machine learning environments, but it appears incremental as it builds on existing Bayesian frameworks.
The paper tackles the problem of learning appropriate bias for an environment of related tasks from a Bayesian perspective, showing that a learner can use Bayesian inference to learn the true prior by sampling from an objective prior, with bounds indicating that sampling multiple tasks is advantageous when prior knowledge is limited and dimensionality is small.
In this paper the problem of learning appropriate bias for an environment of related tasks is examined from a Bayesian perspective. The environment of related tasks is shown to be naturally modelled by the concept of an {\em objective} prior distribution. Sampling from the objective prior corresponds to sampling different learning tasks from the environment. It is argued that for many common machine learning problems, although we don't know the true (objective) prior for the problem, we do have some idea of a set of possible priors to which the true prior belongs. It is shown that under these circumstances a learner can use Bayesian inference to learn the true prior by sampling from the objective prior. Bounds are given on the amount of information required to learn a task when it is simultaneously learnt with several other tasks. The bounds show that if the learner has little knowledge of the true prior, and the dimensionality of the true prior is small, then sampling multiple tasks is highly advantageous.