Optimization with Momentum: Dynamical, Control-Theoretic, and Symplectic Perspectives
This work provides theoretical insights for researchers in optimization and machine learning, though it appears incremental as it builds on existing momentum-based methods with a new analytical perspective.
The paper tackles the problem of analyzing convergence rates for momentum-based optimization algorithms by using dynamical systems theory, obtaining closed-form expressions that relate algorithm parameters to convergence rates across various settings. The result includes a rigorous explanation of why symplectic discretization schemes are important for these algorithms and a characterization of accelerated convergence.
We analyze the convergence rate of various momentum-based optimization algorithms from a dynamical systems point of view. Our analysis exploits fundamental topological properties, such as the continuous dependence of iterates on their initial conditions, to provide a simple characterization of convergence rates. In many cases, closed-form expressions are obtained that relate algorithm parameters to the convergence rate. The analysis encompasses discrete time and continuous time, as well as time-invariant and time-variant formulations, and is not limited to a convex or Euclidean setting. In addition, the article rigorously establishes why symplectic discretization schemes are important for momentum-based optimization algorithms, and provides a characterization of algorithms that exhibit accelerated convergence.