AI-SARAH: Adaptive and Implicit Stochastic Recursive Gradient Methods
This work addresses the practical issue of manual hyperparameter tuning for researchers and practitioners in machine learning, though it is incremental as it builds on the existing SARAH method.
The paper tackles the problem of tuning step-sizes in stochastic gradient methods by introducing AI-SARAH, a variant that adaptively adjusts step-size based on local geometry, resulting in strong performance compared to classical and state-of-the-art first-order methods in convex machine learning problems.
We present AI-SARAH, a practical variant of SARAH. As a variant of SARAH, this algorithm employs the stochastic recursive gradient yet adjusts step-size based on local geometry. AI-SARAH implicitly computes step-size and efficiently estimates local Lipschitz smoothness of stochastic functions. It is fully adaptive, tune-free, straightforward to implement, and computationally efficient. We provide technical insight and intuitive illustrations on its design and convergence. We conduct extensive empirical analysis and demonstrate its strong performance compared with its classical counterparts and other state-of-the-art first-order methods in solving convex machine learning problems.