Graph Reparameterizations for Enabling 1000+ Monte Carlo Iterations in Bayesian Deep Neural Networks
This addresses the scalability problem in uncertainty estimation for deep learning practitioners, enabling more efficient and effective Bayesian methods in real-world applications like computer vision.
The paper tackles the computational expense of Monte Carlo sampling for uncertainty estimation in Bayesian deep neural networks by identifying probability families that allow computation graphs to be independent or weakly dependent on sample count, enabling over 1000 iterations and improving confident accuracy, training stability, memory usage, and training time for large vision architectures.
Uncertainty estimation in deep models is essential in many real-world applications and has benefited from developments over the last several years. Recent evidence suggests that existing solutions dependent on simple Gaussian formulations may not be sufficient. However, moving to other distributions necessitates Monte Carlo (MC) sampling to estimate quantities such as the KL divergence: it could be expensive and scales poorly as the dimensions of both the input data and the model grow. This is directly related to the structure of the computation graph, which can grow linearly as a function of the number of MC samples needed. Here, we construct a framework to describe these computation graphs, and identify probability families where the graph size can be independent or only weakly dependent on the number of MC samples. These families correspond directly to large classes of distributions. Empirically, we can run a much larger number of iterations for MC approximations for larger architectures used in computer vision with gains in performance measured in confident accuracy, stability of training, memory and training time.