Probabilistic Meta-Representations Of Neural Networks
This work addresses a bottleneck in Bayesian neural networks for researchers in probabilistic machine learning, offering a novel approach to improve model expressiveness and regularization.
The paper tackles the problem of weak priors and independent weight distributions in Bayesian neural networks by introducing a richer prior where units are represented by latent variables, enabling correlations between related weights and acting as a complexity regularizer.
Existing Bayesian treatments of neural networks are typically characterized by weak prior and approximate posterior distributions according to which all the weights are drawn independently. Here, we consider a richer prior distribution in which units in the network are represented by latent variables, and the weights between units are drawn conditionally on the values of the collection of those variables. This allows rich correlations between related weights, and can be seen as realizing a function prior with a Bayesian complexity regularizer ensuring simple solutions. We illustrate the resulting meta-representations and representations, elucidating the power of this prior.