Bayesian Learning for Neural Networks: an algorithmic survey
It addresses the complexity and limited adoption of Bayesian learning in neural networks by offering an accessible guide for researchers and practitioners, but it is incremental as it synthesizes existing methods.
This survey introduces Bayesian learning for neural networks from a practical-algorithmic perspective, covering standard and recent inference approaches including Variational Inference, Natural gradients, and manifold optimization, and provides pseudo-codes for implementation.
The last decade witnessed a growing interest in Bayesian learning. Yet, the technicality of the topic and the multitude of ingredients involved therein, besides the complexity of turning theory into practical implementations, limit the use of the Bayesian learning paradigm, preventing its widespread adoption across different fields and applications. This self-contained survey engages and introduces readers to the principles and algorithms of Bayesian Learning for Neural Networks. It provides an introduction to the topic from an accessible, practical-algorithmic perspective. Upon providing a general introduction to Bayesian Neural Networks, we discuss and present both standard and recent approaches for Bayesian inference, with an emphasis on solutions relying on Variational Inference and the use of Natural gradients. We also discuss the use of manifold optimization as a state-of-the-art approach to Bayesian learning. We examine the characteristic properties of all the discussed methods, and provide pseudo-codes for their implementation, paying attention to practical aspects, such as the computation of the gradients.