Sinkhorn Natural Gradient for Generative Models
This work addresses training challenges in generative models, particularly GANs, by introducing a novel natural gradient method with improved computational efficiency, though it appears incremental as it builds on existing natural gradient approaches.
The authors tackled the problem of training generative adversarial networks by proposing the Sinkhorn Natural Gradient (SiNG) algorithm, which acts as a steepest descent method on probability space using Sinkhorn divergence, and demonstrated its efficiency and efficacy in experiments compared to state-of-the-art SGD-type solvers.
We consider the problem of minimizing a functional over a parametric family of probability measures, where the parameterization is characterized via a push-forward structure. An important application of this problem is in training generative adversarial networks. In this regard, we propose a novel Sinkhorn Natural Gradient (SiNG) algorithm which acts as a steepest descent method on the probability space endowed with the Sinkhorn divergence. We show that the Sinkhorn information matrix (SIM), a key component of SiNG, has an explicit expression and can be evaluated accurately in complexity that scales logarithmically with respect to the desired accuracy. This is in sharp contrast to existing natural gradient methods that can only be carried out approximately. Moreover, in practical applications when only Monte-Carlo type integration is available, we design an empirical estimator for SIM and provide the stability analysis. In our experiments, we quantitatively compare SiNG with state-of-the-art SGD-type solvers on generative tasks to demonstrate its efficiency and efficacy of our method.