Judging Adam: Studying the Performance of Optimization Methods on ML4SE Tasks
This work addresses the lack of optimizer benchmarks in ML for software engineering, suggesting a shift from Adam to RAdam for code-related tasks, though it is incremental as it applies existing methods to a new domain.
The study evaluated optimizer performance on deep learning models for source code tasks, finding that optimizer choice can cause up to two-fold score differences in model quality, with RAdam (and its Lookahead variant) consistently performing best.
Solving a problem with a deep learning model requires researchers to optimize the loss function with a certain optimization method. The research community has developed more than a hundred different optimizers, yet there is scarce data on optimizer performance in various tasks. In particular, none of the benchmarks test the performance of optimizers on source code-related problems. However, existing benchmark data indicates that certain optimizers may be more efficient for particular domains. In this work, we test the performance of various optimizers on deep learning models for source code and find that the choice of an optimizer can have a significant impact on the model quality, with up to two-fold score differences between some of the relatively well-performing optimizers. We also find that RAdam optimizer (and its modification with the Lookahead envelope) is the best optimizer that almost always performs well on the tasks we consider. Our findings show a need for a more extensive study of the optimizers in code-related tasks, and indicate that the ML4SE community should consider using RAdam instead of Adam as the default optimizer for code-related deep learning tasks.