On Empirical Comparisons of Optimizers for Deep Learning
This work addresses the problem of fair benchmarking for optimizers in deep learning, highlighting methodological flaws in empirical comparisons, which is incremental but important for researchers and practitioners.
The paper demonstrates that optimizer comparisons in deep learning are highly sensitive to hyperparameter tuning protocols, showing that results can be contradicted by changing search spaces, and finds that adaptive gradient methods never underperform simpler ones like momentum or gradient descent.
Selecting an optimizer is a central step in the contemporary deep learning pipeline. In this paper, we demonstrate the sensitivity of optimizer comparisons to the hyperparameter tuning protocol. Our findings suggest that the hyperparameter search space may be the single most important factor explaining the rankings obtained by recent empirical comparisons in the literature. In fact, we show that these results can be contradicted when hyperparameter search spaces are changed. As tuning effort grows without bound, more general optimizers should never underperform the ones they can approximate (i.e., Adam should never perform worse than momentum), but recent attempts to compare optimizers either assume these inclusion relationships are not practically relevant or restrict the hyperparameters in ways that break the inclusions. In our experiments, we find that inclusion relationships between optimizers matter in practice and always predict optimizer comparisons. In particular, we find that the popular adaptive gradient methods never underperform momentum or gradient descent. We also report practical tips around tuning often ignored hyperparameters of adaptive gradient methods and raise concerns about fairly benchmarking optimizers for neural network training.