Impact of Parameter Sparsity on Stochastic Gradient MCMC Methods for Bayesian Deep Learning
This addresses the trade-off between computational efficiency and predictive performance in Bayesian deep learning, offering a practical solution for researchers and practitioners, though it is incremental in nature.
The paper tackles the computational and storage costs of Bayesian deep learning by exploring sparse network structures with stochastic gradient MCMC, finding that randomly selected substructures can match the performance of advanced pruning methods while significantly reducing training times.
Bayesian methods hold significant promise for improving the uncertainty quantification ability and robustness of deep neural network models. Recent research has seen the investigation of a number of approximate Bayesian inference methods for deep neural networks, building on both the variational Bayesian and Markov chain Monte Carlo (MCMC) frameworks. A fundamental issue with MCMC methods is that the improvements they enable are obtained at the expense of increased computation time and model storage costs. In this paper, we investigate the potential of sparse network structures to flexibly trade-off model storage costs and inference run time against predictive performance and uncertainty quantification ability. We use stochastic gradient MCMC methods as the core Bayesian inference method and consider a variety of approaches for selecting sparse network structures. Surprisingly, our results show that certain classes of randomly selected substructures can perform as well as substructures derived from state-of-the-art iterative pruning methods while drastically reducing model training times.