LyAm: Robust Non-Convex Optimization for Stable Learning in Noisy Environments
This addresses the issue of robust optimization for deep learning practitioners, particularly in noisy environments, but it is incremental as it builds upon existing optimizers like Adam.
The paper tackles the problem of noisy gradients and unstable convergence in training deep neural networks for computer vision tasks by proposing LyAm, a novel optimizer that integrates Adam with Lyapunov-based stability mechanisms, resulting in improved accuracy, convergence speed, and stability on datasets like CIFAR-10 and CIFAR-100.
Training deep neural networks, particularly in computer vision tasks, often suffers from noisy gradients and unstable convergence, which hinder performance and generalization. In this paper, we propose LyAm, a novel optimizer that integrates Adam's adaptive moment estimation with Lyapunov-based stability mechanisms. LyAm dynamically adjusts the learning rate using Lyapunov stability theory to enhance convergence robustness and mitigate training noise. We provide a rigorous theoretical framework proving the convergence guarantees of LyAm in complex, non-convex settings. Extensive experiments on like as CIFAR-10 and CIFAR-100 show that LyAm consistently outperforms state-of-the-art optimizers in terms of accuracy, convergence speed, and stability, establishing it as a strong candidate for robust deep learning optimization.