Calibrated Sampling-Free Uncertainty Estimation in Bayesian Deep Learning
For practitioners deploying deep learning in high-stakes applications, CVP provides an efficient way to obtain calibrated uncertainty estimates without the computational burden of sampling.
The paper tackles the problem of expensive Monte Carlo sampling for uncertainty estimation in Bayesian deep learning. It proposes Calibrated Variance Propagation (CVP), which achieves comparable uncertainty estimates to MC sampling at a fraction of the cost, improving coverage at 0.5% risk from 8.2% to 14.6% on NLVR2 and from 2.6% to 10.8% on VQAv2.
Modern deep learning models remain notoriously prone to overconfidence, limiting their reliability in high-stakes applications. Bayesian methods aim to counter this by learning a distribution over model parameters, and recent advances now make this feasible for large-scale architectures at costs comparable to AdamW. However, a challenge remains at test time: predictions must be averaged across many forward passes with weights sampled from the posterior, which is prohibitively expensive. Variance propagation offers an efficient alternative, computing layer-wise analytical approximations of uncertainty in a single forward pass. While such techniques are effective for MLPs, their extension to modern architectures remains challenging, due to increased depth and diversity of layer types. To fill this gap, we propose Calibrated Variance Propagation (CVP), which introduces a new propagation method for normalization layers, combines it with recent techniques for handling activation functions, and absorbs residual error through a light calibration step. CVP yields comparably accurate uncertainty estimates to MC sampling across transformers and CNNs, at a fraction of the cost. Against prior variance propagation work, CVP improves coverage at $0.5\%$ risk from $8.2\%$ to $14.6\%$ with BEiT-3 on Visual Reasoning (NLVR2) and from $2.6\%$ to $10.8\%$ with ViLT on VQAv2, with gains extending to convolutional architectures.