HyperbolicLR: Epoch insensitive learning rate scheduler
This addresses the problem of inconsistent learning curves in deep learning optimization for researchers and practitioners, particularly in resource-constrained scenarios, but it is incremental as it builds on existing scheduler concepts.
The study tackled the epoch sensitivity problem in learning rate schedulers by proposing HyperbolicLR and ExpHyperbolicLR, which leverage hyperbolic curves to maintain stable learning curves across varying epochs, achieving more consistent performance improvements than conventional methods in tasks like image classification and time series forecasting.
This study proposes two novel learning rate schedulers -- Hyperbolic Learning Rate Scheduler (HyperbolicLR) and Exponential Hyperbolic Learning Rate Scheduler (ExpHyperbolicLR) -- to address the epoch sensitivity problem that often causes inconsistent learning curves in conventional methods. By leveraging the asymptotic behavior of hyperbolic curves, the proposed schedulers maintain more stable learning curves across varying epoch settings. Specifically, HyperbolicLR applies this property directly in the epoch-learning rate space, while ExpHyperbolicLR extends it to an exponential space. We first determine optimal hyperparameters for each scheduler on a small number of epochs, fix these hyperparameters, and then evaluate performance as the number of epochs increases. Experimental results on various deep learning tasks (e.g., image classification, time series forecasting, and operator learning) demonstrate that both HyperbolicLR and ExpHyperbolicLR achieve more consistent performance improvements than conventional schedulers as training duration grows. These findings suggest that our hyperbolic-based schedulers offer a more robust and efficient approach to deep network optimization, particularly in scenarios constrained by computational resources or time.