Accelerated Parameter-Free Stochastic Optimization
This addresses the need for more robust and accessible optimization algorithms in machine learning, though it appears incremental as it builds on existing methods like UniXGrad and DoG.
The paper tackles the problem of stochastic convex optimization by proposing a parameter-free method that achieves near-optimal rates without requiring prior knowledge of key parameters like the initial distance to optimality, and it shows consistent performance in experiments on convex problems.
We propose a method that achieves near-optimal rates for smooth stochastic convex optimization and requires essentially no prior knowledge of problem parameters. This improves on prior work which requires knowing at least the initial distance to optimality d0. Our method, U-DoG, combines UniXGrad (Kavis et al., 2019) and DoG (Ivgi et al., 2023) with novel iterate stabilization techniques. It requires only loose bounds on d0 and the noise magnitude, provides high probability guarantees under sub-Gaussian noise, and is also near-optimal in the non-smooth case. Our experiments show consistent, strong performance on convex problems and mixed results on neural network training.