Ensemble Model Patching: A Parameter-Efficient Variational Bayesian Neural Network
This addresses the problem of making Bayesian neural networks more practical for machine learning practitioners by reducing overhead while maintaining performance with batch normalization.
The paper tackled the high parameter and implementation overhead of variational Bayesian neural networks by introducing a general variational family that includes dropout as a special case, with specific methods that improve predictive accuracy and achieve near-perfect calibration on a ResNet-18 trained with ImageNet.
Two main obstacles preventing the widespread adoption of variational Bayesian neural networks are the high parameter overhead that makes them infeasible on large networks, and the difficulty of implementation, which can be thought of as "programming overhead." MC dropout [Gal and Ghahramani, 2016] is popular because it sidesteps these obstacles. Nevertheless, dropout is often harmful to model performance when used in networks with batch normalization layers [Li et al., 2018], which are an indispensable part of modern neural networks. We construct a general variational family for ensemble-based Bayesian neural networks that encompasses dropout as a special case. We further present two specific members of this family that work well with batch normalization layers, while retaining the benefits of low parameter and programming overhead, comparable to non-Bayesian training. Our proposed methods improve predictive accuracy and achieve almost perfect calibration on a ResNet-18 trained with ImageNet.