On the Natural Gradient of the Evidence Lower Bound
This provides a theoretical insight for generative machine learning, but it is incremental as it builds on existing ELBO and natural gradient concepts.
The paper shows that the natural gradient of the evidence lower bound (ELBO) gap vanishes in unconstrained optimization, making ELBO maximization equivalent to minimizing Kullback-Leibler divergence, and extends this to constrained models using a geometric condition.
This article studies the Fisher-Rao gradient, also referred to as the natural gradient, of the evidence lower bound (ELBO) which plays a central role in generative machine learning. It reveals that the gap between the evidence and its lower bound, the ELBO, has essentially a vanishing natural gradient within unconstrained optimization. As a result, maximization of the ELBO is equivalent to minimization of the Kullback-Leibler divergence from a target distribution, the primary objective function of learning. Building on this insight, we derive a condition under which this equivalence persists even when optimization is constrained to a model. This condition yields a geometric characterization, which we formalize through the notion of a cylindrical model.