A stochastic first-order method with multi-extrapolated momentum for highly smooth unconstrained optimization
This work addresses optimization efficiency for machine learning and scientific computing by accelerating convergence in stochastic settings with high-order smoothness, representing an incremental advance in algorithm design.
The paper tackles the problem of unconstrained stochastic optimization for highly smooth objective functions by proposing a new stochastic first-order method with multi-extrapolated momentum, achieving a sample complexity of σ(ε^{-(3p+1)/p}) for finding a point with expected gradient norm at most ε, which improves upon prior results without requiring mean-squared smoothness.
In this paper, we consider an unconstrained stochastic optimization problem where the objective function exhibits high-order smoothness. Specifically, we propose a new stochastic first-order method (SFOM) with multi-extrapolated momentum, in which multiple extrapolations are performed in each iteration, followed by a momentum update based on these extrapolations. We demonstrate that the proposed SFOM can accelerate optimization by exploiting the high-order smoothness of the objective function $f$. Assuming that the $p$th-order derivative of $f$ is Lipschitz continuous for some $p\ge2$, and under additional mild assumptions, we establish that our method achieves a sample complexity of $\widetilde{\mathcal{O}}(ε^{-(3p+1)/p})$ for finding a point $x$ such that $\mathbb{E}[\|\nabla f(x)\|]\leε$. To the best of our knowledge, this is the first SFOM to leverage arbitrary-order smoothness of the objective function for acceleration, resulting in a sample complexity that improves upon the best-known results without assuming the mean-squared smoothness condition. Preliminary numerical experiments validate the practical performance of our method and support our theoretical findings.