A Bias-Correction Decentralized Stochastic Gradient Algorithm with Momentum Acceleration
This work addresses efficiency and bias issues in distributed machine learning for large-scale data processing, representing an incremental improvement with novel theoretical analysis.
The paper tackles the problem of data heterogeneity and network sparsity in distributed stochastic optimization by proposing the Exact-Diffusion with Momentum (EDM) algorithm, which achieves sub-linear convergence to a neighborhood of the optimal solution for non-convex functions and linear convergence under the Polyak-Lojasiewicz condition.
Distributed stochastic optimization algorithms can simultaneously process large-scale datasets, significantly accelerating model training. However, their effectiveness is often hindered by the sparsity of distributed networks and data heterogeneity. In this paper, we propose a momentum-accelerated distributed stochastic gradient algorithm, termed Exact-Diffusion with Momentum (EDM), which mitigates the bias from data heterogeneity and incorporates momentum techniques commonly used in deep learning to enhance convergence rate. Our theoretical analysis demonstrates that the EDM algorithm converges sub-linearly to the neighborhood of the optimal solution, the radius of which is irrespective of data heterogeneity, when applied to non-convex objective functions; under the Polyak-Lojasiewicz condition, which is a weaker assumption than strong convexity, it converges linearly to the target region. Our analysis techniques employed to handle momentum in complex distributed parameter update structures yield a sufficiently tight convergence upper bound, offering a new perspective for the theoretical analysis of other momentum-based distributed algorithms.