Model Diffusion for Certifiable Few-shot Transfer Learning
This work addresses the problem of providing generalization guarantees for few-shot transfer learning, which is crucial for high-importance applications that require certifiable accuracy, such as those with ethical or legal implications.
The authors tackled the problem of certifiable few-shot transfer learning and achieved non-trivial generalization guarantees, outperforming existing approaches that result in vacuous bounds in the low-shot regime. Their method provides tighter risk certificates by confining the model hypothesis to a finite set of parameter-efficient fine-tuning samples.
In contemporary deep learning, a prevalent and effective workflow for solving low-data problems is adapting powerful pre-trained foundation models (FMs) to new tasks via parameter-efficient fine-tuning (PEFT). However, while empirically effective, the resulting solutions lack generalisation guarantees to certify their accuracy - which may be required for ethical or legal reasons prior to deployment in high-importance applications. In this paper we develop a novel transfer learning approach that is designed to facilitate non-vacuous learning theoretic generalisation guarantees for downstream tasks, even in the low-shot regime. Specifically, we first use upstream tasks to train a distribution over PEFT parameters. We then learn the downstream task by a sample-and-evaluate procedure -- sampling plausible PEFTs from the trained diffusion model and selecting the one with the highest likelihood on the downstream data. Crucially, this confines our model hypothesis to a finite set of PEFT samples. In contrast to the typical continuous hypothesis spaces of neural network weights, this facilitates tighter risk certificates. We instantiate our bound and show non-trivial generalization guarantees compared to existing learning approaches which lead to vacuous bounds in the low-shot regime.