Information Geometry of Variational Bayes
This work clarifies foundational links between two fields to encourage more interdisciplinary research, but it is incremental as it builds on existing concepts.
The paper explores the connection between information geometry and variational Bayes, showing that variational Bayes solutions often require natural gradients, which simplifies Bayes' rule and enables large-scale implementations for models like large language models.
We highlight a fundamental connection between information geometry and variational Bayes (VB) and discuss its consequences for machine learning. Under certain conditions, a VB solution always requires estimation or computation of natural gradients. We show several consequences of this fact by using the natural-gradient descent algorithm of Khan and Rue (2023) called the Bayesian Learning Rule (BLR). These include (i) a simplification of Bayes' rule as addition of natural gradients, (ii) a generalization of quadratic surrogates used in gradient-based methods, and (iii) a large-scale implementation of VB algorithms for large language models. Neither the connection nor its consequences are new but we further emphasize the common origins of the two fields of information geometry and Bayes with a hope to facilitate more work at the intersection of the two fields.