Fast Optimizer Benchmark
It provides a convenient benchmarking tool for researchers and developers working on optimizer development, but it is incremental as it builds on existing benchmarking practices.
The paper introduces the Fast Optimizer Benchmark (FOB), a tool for evaluating deep learning optimizers across domains like computer vision and NLP, featuring user-friendly configurations and integration with HPO tools.
In this paper, we present the Fast Optimizer Benchmark (FOB), a tool designed for evaluating deep learning optimizers during their development. The benchmark supports tasks from multiple domains such as computer vision, natural language processing, and graph learning. The focus is on convenient usage, featuring human-readable YAML configurations, SLURM integration, and plotting utilities. FOB can be used together with existing hyperparameter optimization (HPO) tools as it handles training and resuming of runs. The modular design enables integration into custom pipelines, using it simply as a collection of tasks. We showcase an optimizer comparison as a usage example of our tool. FOB can be found on GitHub: https://github.com/automl/FOB.