Whiplash Gradient Descent Dynamics
This work addresses optimization challenges for researchers and practitioners in machine learning and applied mathematics, presenting an incremental improvement with novel analysis techniques.
The paper tackles the problem of finding minima of cost functions in finite-dimensional settings by proposing Whiplash Inertial Gradient dynamics, a closed-loop optimization method, and demonstrates polynomial and exponential convergence rates for quadratic cost functions.
In this paper, we propose the Whiplash Inertial Gradient dynamics, a closed-loop optimization method that utilises gradient information, to find the minima of a cost function in finite-dimensional settings. We introduce the symplectic asymptotic convergence analysis for the Whiplash system for convex functions. We also introduce relaxation sequences to explain the non-classical nature of the algorithm and an exploring heuristic variant of the Whiplash algorithm to escape saddle points, deterministically. We study the algorithm's performance for various costs and provide a practical methodology for analyzing convergence rates using integral constraint bounds and a novel Lyapunov rate method. Our results demonstrate polynomial and exponential rates of convergence for quadratic cost functions.