The Physical Systems Behind Optimization Algorithms
This work offers a theoretical framework for understanding optimization dynamics, which is incremental as it extends existing analysis methods to broader conditions.
The paper tackles the problem of analyzing optimization algorithms in machine learning by using differential equations to provide physics-based insights, resulting in a unified framework applicable to general algorithms and nonconvex conditions like Polyak-Łojasiewicz.
We use differential equations based approaches to provide some {\it \textbf{physics}} insights into analyzing the dynamics of popular optimization algorithms in machine learning. In particular, we study gradient descent, proximal gradient descent, coordinate gradient descent, proximal coordinate gradient, and Newton's methods as well as their Nesterov's accelerated variants in a unified framework motivated by a natural connection of optimization algorithms to physical systems. Our analysis is applicable to more general algorithms and optimization problems {\it \textbf{beyond}} convexity and strong convexity, e.g. Polyak-Łojasiewicz and error bound conditions (possibly nonconvex).