Mirrorless Mirror Descent: A Natural Derivation of Mirror Descent
This provides a theoretical insight into optimization methods for researchers in machine learning and optimization, but it is incremental as it builds on existing Mirror Descent and Natural Gradient Descent frameworks.
The paper tackles the problem of deriving Mirror Descent from a primal perspective by showing it as a partial discretization of gradient flow on a Riemannian manifold with a Hessian metric, contrasting it with Natural Gradient Descent, and generalizing it to non-Hessian metrics without a dual.
We present a primal only derivation of Mirror Descent as a "partial" discretization of gradient flow on a Riemannian manifold where the metric tensor is the Hessian of the Mirror Descent potential. We contrast this discretization to Natural Gradient Descent, which is obtained by a "full" forward Euler discretization. This view helps shed light on the relationship between the methods and allows generalizing Mirror Descent to general Riemannian geometries, even when the metric tensor is {\em not} a Hessian, and thus there is no "dual."