Asynchronous decentralized accelerated stochastic gradient descent
This addresses communication and synchronization bottlenecks in decentralized optimization, offering improved efficiency for distributed machine learning systems, though it appears incremental as it builds on existing accelerated and decentralized methods.
The paper tackles decentralized stochastic optimization by introducing an asynchronous decentralized accelerated stochastic gradient descent method, achieving communication complexities of O(1/ε) for general convex and O(1/√ε) for strongly convex problems, and sampling complexities of O(1/ε²) and O(1/ε), respectively.
In this work, we introduce an asynchronous decentralized accelerated stochastic gradient descent type of method for decentralized stochastic optimization, considering communication and synchronization are the major bottlenecks. We establish $\mathcal{O}(1/ε)$ (resp., $\mathcal{O}(1/\sqrtε)$) communication complexity and $\mathcal{O}(1/ε^2)$ (resp., $\mathcal{O}(1/ε)$) sampling complexity for solving general convex (resp., strongly convex) problems.