Better Parameter-free Stochastic Optimization with ODE Updates for Coin-Betting
This addresses the practical deployment challenge of parameter-free optimization for machine learning practitioners by closing the empirical gap with tuned methods.
The paper tackles the empirical performance gap between parameter-free stochastic gradient descent (PFSGD) and tuned SGD by introducing a new parameter-free algorithm based on continuous-time Coin-Betting with ODE-derived updates. The result shows this algorithm outperforms methods with 'best default' learning rates and nearly matches finely tuned baselines without requiring tuning.
Parameter-free stochastic gradient descent (PFSGD) algorithms do not require setting learning rates while achieving optimal theoretical performance. In practical applications, however, there remains an empirical gap between tuned stochastic gradient descent (SGD) and PFSGD. In this paper, we close the empirical gap with a new parameter-free algorithm based on continuous-time Coin-Betting on truncated models. The new update is derived through the solution of an Ordinary Differential Equation (ODE) and solved in a closed form. We show empirically that this new parameter-free algorithm outperforms algorithms with the "best default" learning rates and almost matches the performance of finely tuned baselines without anything to tune.