Sampling-free Variational Inference for Neural Networks with Multiplicative Activation Noise
This work addresses the need for efficient uncertainty estimation in neural networks for safety-critical domains, offering a parameter-efficient method that is incremental over existing sampling-free approaches.
The paper tackled the problem of inefficient parameterization in sampling-free variational inference for Bayesian neural networks by proposing a new posterior approximation based on multiplicative Gaussian activation noise, achieving competitive results on standard regression and scaling to large-scale image classification tasks like ImageNet.
To adopt neural networks in safety critical domains, knowing whether we can trust their predictions is crucial. Bayesian neural networks (BNNs) provide uncertainty estimates by averaging predictions with respect to the posterior weight distribution. Variational inference methods for BNNs approximate the intractable weight posterior with a tractable distribution, yet mostly rely on sampling from the variational distribution during training and inference. Recent sampling-free approaches offer an alternative, but incur a significant parameter overhead. We here propose a more efficient parameterization of the posterior approximation for sampling-free variational inference that relies on the distribution induced by multiplicative Gaussian activation noise. This allows us to combine parameter efficiency with the benefits of sampling-free variational inference. Our approach yields competitive results for standard regression problems and scales well to large-scale image classification tasks including ImageNet.