Amortising Inference and Meta-Learning Priors in Neural Networks
This addresses a core problem in Bayesian deep learning for researchers and practitioners by providing a method to specify priors, though it is incremental in combining existing fields.
The paper tackles the challenge of lacking prior beliefs in Bayesian deep learning by learning a weights prior from multiple datasets through amortised variational inference, enabling Bayesian neural networks to function as generative models and perform meta-learning with minimal data.
One of the core facets of Bayesianism is in the updating of prior beliefs in light of new evidence$\text{ -- }$so how can we maintain a Bayesian approach if we have no prior beliefs in the first place? This is one of the central challenges in the field of Bayesian deep learning, where it is not clear how to represent beliefs about a prediction task by prior distributions over model parameters. Bridging the fields of Bayesian deep learning and probabilistic meta-learning, we introduce a way to $\textit{learn}$ a weights prior from a collection of datasets by introducing a way to perform per-dataset amortised variational inference. The model we develop can be viewed as a neural process whose latent variable is the set of weights of a BNN and whose decoder is the neural network parameterised by a sample of the latent variable itself. This unique model allows us to study the behaviour of Bayesian neural networks under well-specified priors, use Bayesian neural networks as flexible generative models, and perform desirable but previously elusive feats in neural processes such as within-task minibatching or meta-learning under extreme data-starvation.