Do Bayesian Neural Networks Need To Be Fully Stochastic?
This work addresses the computational inefficiency of Bayesian neural networks for practitioners, showing that full stochasticity may be unnecessary, which is an incremental improvement.
The paper investigates whether Bayesian neural networks require full stochasticity in all parameters, finding that only a small number of stochastic biases are needed for universal probabilistic prediction, with empirical results showing partially stochastic networks match or outperform fully stochastic ones on multiple datasets.
We investigate the benefit of treating all the parameters in a Bayesian neural network stochastically and find compelling theoretical and empirical evidence that this standard construction may be unnecessary. To this end, we prove that expressive predictive distributions require only small amounts of stochasticity. In particular, partially stochastic networks with only $n$ stochastic biases are universal probabilistic predictors for $n$-dimensional predictive problems. In empirical investigations, we find no systematic benefit of full stochasticity across four different inference modalities and eight datasets; partially stochastic networks can match and sometimes even outperform fully stochastic networks, despite their reduced memory costs.