DoG is SGD's Best Friend: A Parameter-Free Dynamic Step Size Schedule
This provides a tuning-free alternative for SGD users, though it is incremental as it builds on existing optimization methods.
The authors tackled the problem of tuning SGD step sizes by introducing a parameter-free dynamic schedule called DoG, which uses distance from the initial point and gradient norms, and showed it performs close to tuned SGD on vision and language tasks.
We propose a tuning-free dynamic SGD step size formula, which we call Distance over Gradients (DoG). The DoG step sizes depend on simple empirical quantities (distance from the initial point and norms of gradients) and have no ``learning rate'' parameter. Theoretically, we show that a slight variation of the DoG formula enjoys strong parameter-free convergence guarantees for stochastic convex optimization assuming only \emph{locally bounded} stochastic gradients. Empirically, we consider a broad range of vision and language transfer learning tasks, and show that DoG's performance is close to that of SGD with tuned learning rate. We also propose a per-layer variant of DoG that generally outperforms tuned SGD, approaching the performance of tuned Adam. A PyTorch implementation is available at https://github.com/formll/dog