Fast symplectic integrator for Nesterov-type acceleration method
This work addresses the need for more stable and efficient optimization algorithms in machine learning, though it appears incremental as it builds on existing geometric frameworks.
The paper tackled the problem of improving the convergence rate of Nesterov's accelerated gradient method by proposing explicit stable integrators based on symplectic and contact geometries for a non-autonomous ODE, achieving high convergence rates in numerical experiments.
In this paper, explicit stable integrators based on symplectic and contact geometries are proposed for a non-autonomous ordinarily differential equation (ODE) found in improving convergence rate of Nesterov's accelerated gradient method. Symplectic geometry is known to be suitable for describing Hamiltonian mechanics, and contact geometry is known as an odd-dimensional counterpart of symplectic geometry. Moreover, a procedure, called symplectization, is a known way to construct a symplectic manifold from a contact manifold, yielding Hamiltonian systems from contact ones. It is found in this paper that a previously investigated non-autonomous ODE can be written as a contact Hamiltonian system. Then, by symplectization of a non-autonomous contact Hamiltonian vector field expressing the non-autonomous ODE, novel symplectic integrators are derived. Because the proposed symplectic integrators preserve hidden symplectic and contact structures in the ODE, they should be more stable than the Runge-Kutta method. Numerical experiments demonstrate that, as expected, the second-order symplectic integrator is stable and high convergence rates are achieved.