Pre-Train Your Loss: Easy Bayesian Transfer Learning with Informative Priors
This work addresses the challenge of improving transfer learning efficiency for practitioners in machine learning, offering a modular drop-in replacement for standard pre-training strategies.
The paper tackles the problem of limited information transfer in deep learning by proposing to learn informative posteriors from source tasks to use as priors for downstream tasks, resulting in significant performance gains and more data-efficient learning on classification and segmentation tasks.
Deep learning is increasingly moving towards a transfer learning paradigm whereby large foundation models are fine-tuned on downstream tasks, starting from an initialization learned on the source task. But an initialization contains relatively little information about the source task. Instead, we show that we can learn highly informative posteriors from the source task, through supervised or self-supervised approaches, which then serve as the basis for priors that modify the whole loss surface on the downstream task. This simple modular approach enables significant performance gains and more data-efficient learning on a variety of downstream classification and segmentation tasks, serving as a drop-in replacement for standard pre-training strategies. These highly informative priors also can be saved for future use, similar to pre-trained weights, and stand in contrast to the zero-mean isotropic uninformative priors that are typically used in Bayesian deep learning.