Is Scaling Learned Optimizers Worth It? Evaluating The Value of VeLO's 4000 TPU Months
This challenges the value of large-scale learned optimizers for machine learning practitioners, showing incremental or negative results.
The paper evaluates VeLO, a learned optimizer trained with 4000 TPU months, and finds it does not outperform standard optimizers like Adam in solution quality or speed, and requires problem-specific tuning.
We analyze VeLO (versatile learned optimizer), the largest scale attempt to train a general purpose "foundational" optimizer to date. VeLO was trained on thousands of machine learning tasks using over 4000 TPU months with the goal of producing an optimizer capable of generalizing to new problems while being hyperparameter free, and outperforming industry standards such as Adam. We independently evaluate VeLO on the MLCommons optimizer benchmark suite. We find that, contrary to initial claims: (1) VeLO has a critical hyperparameter that needs problem-specific tuning, (2) VeLO does not necessarily outperform competitors in quality of solution found, and (3) VeLO is not faster than competing optimizers at reducing the training loss. These observations call into question VeLO's generality and the value of the investment in training it.