A closed loop gradient descent algorithm applied to Rosenbrock's function
This work addresses optimization challenges in machine learning and engineering, but it appears incremental as it builds on existing momentum-based gradient methods.
The authors tackled the problem of unconstrained optimization by introducing an adaptive damping technique for an inertial gradient system, applied as a gradient descent algorithm, and demonstrated an improvement on existing momentum-based methods using the non-convex Rosenbrock's function.
We introduce a novel adaptive damping technique for an inertial gradient system which finds application as a gradient descent algorithm for unconstrained optimisation. In an example using the non-convex Rosenbrock's function, we show an improvement on existing momentum-based gradient optimisation methods. Also using Lyapunov stability analysis, we demonstrate the performance of the continuous-time version of the algorithm. Using numerical simulations, we consider the performance of its discrete-time counterpart obtained by using the symplectic Euler method of discretisation.