Break a Lag: Triple Exponential Moving Average for Enhanced Optimization
This addresses optimization inefficiencies for deep learning practitioners, offering a scalable solution with substantial improvements, though it appears incremental as an enhancement to existing EMA-based methods.
The paper tackles the problem of lag in trend identification and suboptimal optimization in deep learning by introducing FAME, a novel optimizer using Triple Exponential Moving Average, which demonstrates superior accuracy and robustness across 30 architectures in computer vision tasks.
The performance of deep learning models is critically dependent on sophisticated optimization strategies. While existing optimizers have shown promising results, many rely on first-order Exponential Moving Average (EMA) techniques, which often limit their ability to track complex gradient trends accurately. This fact can lead to a significant lag in trend identification and suboptimal optimization, particularly in highly dynamic gradient behavior. To address this fundamental limitation, we introduce Fast Adaptive Moment Estimation (FAME), a novel optimization technique that leverages the power of Triple Exponential Moving Average. By incorporating an advanced tracking mechanism, FAME enhances responsiveness to data dynamics, mitigates trend identification lag, and optimizes learning efficiency. Our comprehensive evaluation encompasses different computer vision tasks including image classification, object detection, and semantic segmentation, integrating FAME into 30 distinct architectures ranging from lightweight CNNs to Vision Transformers. Through rigorous benchmarking against state-of-the-art optimizers, FAME demonstrates superior accuracy and robustness. Notably, it offers high scalability, delivering substantial improvements across diverse model complexities, architectures, tasks, and benchmarks.