Radial and Directional Posteriors for Bayesian Neural Networks
This work addresses the challenge of modeling dependencies in Bayesian neural networks for researchers and practitioners, offering an incremental improvement over existing variational methods.
The authors tackled the problem of improving variational inference for Bayesian neural networks by proposing a new variational family that decomposes the posterior into radial and directional components, resulting in better predictive performance and network compression.
We propose a new variational family for Bayesian neural networks. We decompose the variational posterior into two components, where the radial component captures the strength of each neuron in terms of its magnitude; while the directional component captures the statistical dependencies among the weight parameters. The dependencies learned via the directional density provide better modeling performance compared to the widely-used Gaussian mean-field-type variational family. In addition, the strength of input and output neurons learned via the radial density provides a structured way to compress neural networks. Indeed, experiments show that our variational family improves predictive performance and yields compressed networks simultaneously.