Gravilon: Applications of a New Gradient Descent Method to Machine Learning
This work addresses a fundamental optimization challenge in machine learning, but it appears incremental as it builds on existing gradient descent methods with a specific geometric modification.
The authors tackled the problem of improving gradient descent efficiency and accuracy in neural networks by introducing Gravilon, a novel algorithm that modifies step length based on hypersurface geometry, and demonstrated promising experimental results on MNIST digit classification.
Gradient descent algorithms have been used in countless applications since the inception of Newton's method. The explosion in the number of applications of neural networks has re-energized efforts in recent years to improve the standard gradient descent method in both efficiency and accuracy. These methods modify the effect of the gradient in updating the values of the parameters. These modifications often incorporate hyperparameters: additional variables whose values must be specified at the outset of the program. We provide, below, a novel gradient descent algorithm, called Gravilon, that uses the geometry of the hypersurface to modify the length of the step in the direction of the gradient. Using neural networks, we provide promising experimental results comparing the accuracy and efficiency of the Gravilon method against commonly used gradient descent algorithms on MNIST digit classification.