Fine-Tuning Adaptive Stochastic Optimizers: Determining the Optimal Hyperparameter $ε$ via Gradient Magnitude Histogram Analysis
This work addresses the computational inefficiency in hyperparameter tuning for deep learning practitioners, though it is incremental as it focuses on a specific hyperparameter within existing optimizers.
The paper tackles the problem of efficiently tuning the safeguard hyperparameter ε in adaptive stochastic optimizers like Adam, which is often overlooked, by introducing a gradient magnitude histogram framework; this results in a method that reduces the search space for optimal ε by half compared to trial-and-error approaches.
Stochastic optimizers play a crucial role in the successful training of deep neural network models. To achieve optimal model performance, designers must carefully select both model and optimizer hyperparameters. However, this process is frequently demanding in terms of computational resources and processing time. While it is a well-established practice to tune the entire set of optimizer hyperparameters for peak performance, there is still a lack of clarity regarding the individual influence of hyperparameters mislabeled as "low priority", including the safeguard factor $ε$ and decay rate $β$, in leading adaptive stochastic optimizers like the Adam optimizer. In this manuscript, we introduce a new framework based on the empirical probability density function of the loss' gradient magnitude, termed as the "gradient magnitude histogram", for a thorough analysis of adaptive stochastic optimizers and the safeguard hyperparameter $ε$. This framework reveals and justifies valuable relationships and dependencies among hyperparameters in connection to optimal performance across diverse tasks, such as classification, language modeling and machine translation. Furthermore, we propose a novel algorithm using gradient magnitude histograms to automatically estimate a refined and accurate search space for the optimal safeguard hyperparameter $ε$, surpassing the conventional trial-and-error methodology by establishing a worst-case search space that is two times narrower.