Guiding Time-Varying Generative Models with Natural Gradients on Exponential Family Manifold
This work addresses a gap in connecting probabilistic model optimization with generative model training, offering a novel approach for researchers in machine learning and statistics, though it appears incremental in its methodological contribution.
The paper tackles the problem of training time-varying generative models by projecting their evolution onto an exponential family manifold and using natural gradient descent, resulting in efficient approximation of natural gradients without MCMC and validation through experiments.
Optimising probabilistic models is a well-studied field in statistics. However, its connection with the training of generative models remains largely under-explored. In this paper, we show that the evolution of time-varying generative models can be projected onto an exponential family manifold, naturally creating a link between the parameters of a generative model and those of a probabilistic model. We then train the generative model by moving its projection on the manifold according to the natural gradient descent scheme. This approach also allows us to efficiently approximate the natural gradient of the KL divergence without relying on MCMC for intractable models. Furthermore, we propose particle versions of the algorithm, which feature closed-form update rules for any parametric model within the exponential family. Through toy and real-world experiments, we validate the effectiveness of the proposed algorithms. The code of the proposed algorithms can be found at https://github.com/anewgithubname/iNGD.