Structured and Efficient Variational Deep Learning with Matrix Gaussian Posteriors
This work addresses efficiency in variational deep learning for researchers and practitioners, but it is incremental as it builds on existing methods like matrix Gaussians and local reparameterization.
The paper tackles the challenge of efficient variational inference in Bayesian neural networks by introducing a matrix Gaussian posterior that models parameter correlations, achieving computational savings compared to fully factorized posteriors. It demonstrates the approach through experiments, showing improved efficiency without specifying concrete performance numbers.
We introduce a variational Bayesian neural network where the parameters are governed via a probability distribution on random matrices. Specifically, we employ a matrix variate Gaussian \cite{gupta1999matrix} parameter posterior distribution where we explicitly model the covariance among the input and output dimensions of each layer. Furthermore, with approximate covariance matrices we can achieve a more efficient way to represent those correlations that is also cheaper than fully factorized parameter posteriors. We further show that with the "local reprarametrization trick" \cite{kingma2015variational} on this posterior distribution we arrive at a Gaussian Process \cite{rasmussen2006gaussian} interpretation of the hidden units in each layer and we, similarly with \cite{gal2015dropout}, provide connections with deep Gaussian processes. We continue in taking advantage of this duality and incorporate "pseudo-data" \cite{snelson2005sparse} in our model, which in turn allows for more efficient sampling while maintaining the properties of the original model. The validity of the proposed approach is verified through extensive experiments.