ZENITH: Automated Gradient Norm Informed Stochastic Optimization
This addresses the computational and compatibility issues of adaptive optimizers for training deep computer vision models, though it appears incremental as it builds on gradient norm concepts.
The paper tackles the problem of manual learning rate scheduling in deep computer vision by introducing the ZENITH optimizer, which adapts learning rates based on gradient norm evolution. Results show it achieves higher test accuracy in lower wall-clock time across 6 CNN architectures and 6 benchmarks, with superior mAP in object detection, keypoint detection, and instance segmentation on MS COCO.
Training deep computer vision models requires manual oversight or hyperparameter tuning of the learning rate (LR) schedule. While existing adaptive optimizers schedule the LR automatically, they suffer from computational and memory overhead, incompatibility with regularization, and suboptimal LR choices. In this work, we introduce the ZENITH (Zero-overhead Evolution using Norm-Informed Training History) optimizer, which adapts the LR using the temporal evolution of the gradient norm. Image classification experiments spanning 6 CNN architectures and 6 benchmarks demonstrate that ZENITH achieves higher test accuracy in lower wall-clock time than baselines. It also yielded superior mAP in object detection, keypoint detection, and instance segmentation on MS COCO using the R-CNN family of models. Furthermore, its compatibility with regularization enables even better generalization.