Optimal Asynchronous Stochastic Nonconvex Optimization under Heavy-Tailed Noise
This work addresses optimization challenges in distributed systems with noisy and variable conditions, representing an incremental improvement in asynchronous methods.
The paper tackles the problem of asynchronous stochastic nonconvex optimization with heavy-tailed gradient noise and heterogeneous worker computation times by proposing an asynchronous normalized stochastic gradient descent algorithm with momentum, achieving optimal time complexity under bounded pth-order central moment assumptions.
This paper considers the problem of asynchronous stochastic nonconvex optimization with heavy-tailed gradient noise and arbitrarily heterogeneous computation times across workers. We propose an asynchronous normalized stochastic gradient descent algorithm with momentum. The analysis show that our method achieves the optimal time complexity under the assumption of bounded $p$th-order central moment with $p\in(1,2]$. We also provide numerical experiments to show the effectiveness of proposed method.