Angle based dynamic learning rate for gradient descent
This work addresses the optimization challenge in machine learning for classification, offering an incremental improvement over existing state-of-the-art optimizers.
The authors tackled the problem of adaptive learning rate selection in gradient descent for classification tasks by using the angle between current and orthogonal gradients, achieving the highest accuracy on most benchmark datasets with various image classification architectures.
In our work, we propose a novel yet simple approach to obtain an adaptive learning rate for gradient-based descent methods on classification tasks. Instead of the traditional approach of selecting adaptive learning rates via the decayed expectation of gradient-based terms, we use the angle between the current gradient and the new gradient: this new gradient is computed from the direction orthogonal to the current gradient, which further helps us in determining a better adaptive learning rate based on angle history, thereby, leading to relatively better accuracy compared to the existing state-of-the-art optimizers. On a wide variety of benchmark datasets with prominent image classification architectures such as ResNet, DenseNet, EfficientNet, and VGG, we find that our method leads to the highest accuracy in most of the datasets. Moreover, we prove that our method is convergent.