Priors in Bayesian Deep Learning: A Review
This review addresses the problem of improving prior specification for practitioners in Bayesian deep learning, but it is incremental as it synthesizes existing work without introducing new methods.
The paper reviews the critical role of priors in Bayesian deep learning, highlighting that recent models often use vague priors like standard Gaussians, and it provides an overview of different priors and methods for learning them from data for models such as Gaussian processes, variational autoencoders, and Bayesian neural networks.
While the choice of prior is one of the most critical parts of the Bayesian inference workflow, recent Bayesian deep learning models have often fallen back on vague priors, such as standard Gaussians. In this review, we highlight the importance of prior choices for Bayesian deep learning and present an overview of different priors that have been proposed for (deep) Gaussian processes, variational autoencoders, and Bayesian neural networks. We also outline different methods of learning priors for these models from data. We hope to motivate practitioners in Bayesian deep learning to think more carefully about the prior specification for their models and to provide them with some inspiration in this regard.