Tubular Riemannian Laplace Approximations for Bayesian Neural Networks
This work addresses the problem of efficient and reliable uncertainty estimation in deep learning for practitioners, offering a method that bridges single-model efficiency and ensemble-grade reliability, though it builds incrementally on prior Riemannian approximations.
The paper tackles the challenge of approximate Bayesian inference in neural networks by introducing the Tubular Riemannian Laplace (TRL) approximation, which models the posterior as a probabilistic tube to adapt to anisotropic loss surfaces and symmetry groups, achieving excellent calibration on ResNet-18 with CIFAR datasets while reducing training cost to 1/5 of Deep Ensembles.
Laplace approximations are among the simplest and most practical methods for approximate Bayesian inference in neural networks, yet their Euclidean formulation struggles with the highly anisotropic, curved loss surfaces and large symmetry groups that characterize modern deep models. Recent work has proposed Riemannian and geometric Gaussian approximations to adapt to this structure. Building on these ideas, we introduce the Tubular Riemannian Laplace (TRL) approximation. TRL explicitly models the posterior as a probabilistic tube that follows a low-loss valley induced by functional symmetries, using a Fisher/Gauss-Newton metric to separate prior-dominated tangential uncertainty from data-dominated transverse uncertainty. We interpret TRL as a scalable reparametrised Gaussian approximation that utilizes implicit curvature estimates to operate in high-dimensional parameter spaces. Our empirical evaluation on ResNet-18 (CIFAR-10 and CIFAR-100) demonstrates that TRL achieves excellent calibration, matching or exceeding the reliability of Deep Ensembles (in terms of ECE) while requiring only a fraction (1/5) of the training cost. TRL effectively bridges the gap between single-model efficiency and ensemble-grade reliability.