Mechanic: A Learning Rate Tuner
This addresses the challenge of hyperparameter tuning for practitioners in machine learning, though it appears incremental as it builds on existing theoretical reductions.
The paper tackles the problem of automatically tuning the learning rate scale factor for optimization algorithms in deep learning, demonstrating that their method, Mechanic, either closely matches or improves upon manual tuning across various tasks.
We introduce a technique for tuning the learning rate scale factor of any base optimization algorithm and schedule automatically, which we call \textsc{mechanic}. Our method provides a practical realization of recent theoretical reductions for accomplishing a similar goal in online convex optimization. We rigorously evaluate \textsc{mechanic} on a range of large scale deep learning tasks with varying batch sizes, schedules, and base optimization algorithms. These experiments demonstrate that depending on the problem, \textsc{mechanic} either comes very close to, matches or even improves upon manual tuning of learning rates.