Adaptive Friction in Deep Learning: Enhancing Optimizers with Sigmoid and Tanh Function
This work addresses optimization issues in deep learning for researchers and practitioners, offering an incremental improvement by enhancing existing optimizers with adaptive friction coefficients.
The paper tackled challenges like poor generalization and oscillation in adaptive optimizers by introducing sigSignGrad and tanhSignGrad, which integrate adaptive friction coefficients based on Sigmoid and Tanh functions, resulting in improved accuracy and reduced training time on datasets like CIFAR-10, CIFAR-100, and Mini-ImageNet.
Adaptive optimizers are pivotal in guiding the weight updates of deep neural networks, yet they often face challenges such as poor generalization and oscillation issues. To counter these, we introduce sigSignGrad and tanhSignGrad, two novel optimizers that integrate adaptive friction coefficients based on the Sigmoid and Tanh functions, respectively. These algorithms leverage short-term gradient information, a feature overlooked in traditional Adam variants like diffGrad and AngularGrad, to enhance parameter updates and convergence.Our theoretical analysis demonstrates the wide-ranging adjustment capability of the friction coefficient S, which aligns with targeted parameter update strategies and outperforms existing methods in both optimization trajectory smoothness and convergence rate. Extensive experiments on CIFAR-10, CIFAR-100, and Mini-ImageNet datasets using ResNet50 and ViT architectures confirm the superior performance of our proposed optimizers, showcasing improved accuracy and reduced training time. The innovative approach of integrating adaptive friction coefficients as plug-ins into existing optimizers, exemplified by the sigSignAdamW and sigSignAdamP variants, presents a promising strategy for boosting the optimization performance of established algorithms. The findings of this study contribute to the advancement of optimizer design in deep learning.