Walking on the Fiber: A Simple Geometric Approximation for Bayesian Neural Networks
This work addresses the problem of scalable and accurate Bayesian inference in deep learning for practitioners, though it appears incremental as it builds on existing sampling techniques.
The paper tackles the computational intractability of Bayesian Neural Networks for uncertainty quantification by proposing a simple sampling variation that leverages low-dimensional loss minima structure and a deformation model for rapid posterior sampling, achieving competitive posterior approximations with improved scalability compared to recent techniques.
Bayesian Neural Networks provide a principled framework for uncertainty quantification by modeling the posterior distribution of network parameters. However, exact posterior inference is computationally intractable, and widely used approximations like the Laplace method struggle with scalability and posterior accuracy in modern deep networks. In this work, we revisit sampling techniques for posterior exploration, proposing a simple variation tailored to efficiently sample from the posterior in over-parameterized networks by leveraging the low-dimensional structure of loss minima. Building on this, we introduce a model that learns a deformation of the parameter space, enabling rapid posterior sampling without requiring iterative methods. Empirical results demonstrate that our approach achieves competitive posterior approximations with improved scalability compared to recent refinement techniques. These contributions provide a practical alternative for Bayesian inference in deep learning.