BayesAdapter: Being Bayesian, Inexpensively and Reliably, via Bayesian Fine-tuning
This work addresses the practical adoption challenges of BNNs for machine learning practitioners, offering an incremental improvement in efficiency and reliability.
The paper tackles the scalability, accessibility, and reliability issues of Bayesian neural networks (BNNs) by proposing BayesAdapter, a framework that adapts pre-trained deterministic neural networks into variational BNNs via cost-effective fine-tuning, achieving higher quality posteriors and significantly reducing training overheads compared to baselines.
Despite their theoretical appealingness, Bayesian neural networks (BNNs) are left behind in real-world adoption, mainly due to persistent concerns on their scalability, accessibility, and reliability. In this work, we develop the BayesAdapter framework to relieve these concerns. In particular, we propose to adapt pre-trained deterministic NNs to be variational BNNs via cost-effective Bayesian fine-tuning. Technically, we develop a modularized implementation for the learning of variational BNNs, and refurbish the generally applicable exemplar reparameterization trick through exemplar parallelization to efficiently reduce the gradient variance in stochastic variational inference. Based on the lightweight Bayesian learning paradigm, we conduct extensive experiments on a variety of benchmarks, and show that our method can consistently induce posteriors with higher quality than competitive baselines, yet significantly reducing training overheads. Code is available at https://github.com/thudzj/ScalableBDL.