Probabilistic Guarantees of Stochastic Recursive Gradient in Non-Convex Finite Sum Problems
This work addresses the need for reliable probabilistic guarantees in stochastic optimization algorithms for machine learning practitioners, offering an incremental improvement over existing variance-reduced methods.
The paper tackled the problem of providing high-probability guarantees for gradient norm estimators in non-convex finite sum optimization by developing a new Azuma-Hoeffding type bound and proposing Prob-SARAH, a modified version of SARAH, which matches the best in-expectation complexity up to logarithmic factors and shows superior probabilistic performance on real datasets.
This paper develops a new dimension-free Azuma-Hoeffding type bound on summation norm of a martingale difference sequence with random individual bounds. With this novel result, we provide high-probability bounds for the gradient norm estimator in the proposed algorithm Prob-SARAH, which is a modified version of the StochAstic Recursive grAdient algoritHm (SARAH), a state-of-art variance reduced algorithm that achieves optimal computational complexity in expectation for the finite sum problem. The in-probability complexity by Prob-SARAH matches the best in-expectation result up to logarithmic factors. Empirical experiments demonstrate the superior probabilistic performance of Prob-SARAH on real datasets compared to other popular algorithms.