A Formalization of The Natural Gradient Method for General Similarity Measures
This work addresses a theoretical gap in optimization for researchers and practitioners using similarity measures, but it appears incremental as it extends existing natural gradient concepts rather than introducing a new paradigm.
The paper tackles the problem of determining which metric to use for the natural gradient method when minimizing arbitrary similarity measures between distributions, and provides a general framework that derives a metric for the natural gradient, showing overlap with known cases and computing natural gradients in novel frameworks.
In optimization, the natural gradient method is well-known for likelihood maximization. The method uses the Kullback-Leibler divergence, corresponding infinitesimally to the Fisher-Rao metric, which is pulled back to the parameter space of a family of probability distributions. This way, gradients with respect to the parameters respect the Fisher-Rao geometry of the space of distributions, which might differ vastly from the standard Euclidean geometry of the parameter space, often leading to faster convergence. However, when minimizing an arbitrary similarity measure between distributions, it is generally unclear which metric to use. We provide a general framework that, given a similarity measure, derives a metric for the natural gradient. We then discuss connections between the natural gradient method and multiple other optimization techniques in the literature. Finally, we provide computations of the formal natural gradient to show overlap with well-known cases and to compute natural gradients in novel frameworks.