Neograd: Near-Ideal Gradient Descent
This work addresses optimization challenges in machine learning, offering a potentially more efficient gradient descent variant, though it appears incremental as it builds upon existing methods like momentum.
The paper tackled the problems of plateaus in cost function minimization and learning rate adjustment in gradient descent, resulting in NeogradM, a hybrid method that outperforms Adam on several test problems and achieves cost function values smaller by factors such as 10^8.
The purpose of this paper is to improve upon existing variants of gradient descent by solving two problems: (1) removing (or reducing) the plateau that occurs while minimizing the cost function, (2) continually adjusting the learning rate to an "ideal" value. The approach taken is to approximately solve for the learning rate as a function of a trust metric. When this technique is hybridized with momentum, it creates an especially effective gradient descent variant, called NeogradM. It is shown to outperform Adam on several test problems, and can easily reach cost function values that are smaller by a factor of $10^8$, for example.