Local MixVR: Breaking the Communication-Sample Dependence in Distributed Learning
This work addresses the communication bottleneck in scalable distributed learning, offering a new paradigm for communication-efficient training.
Local MixVR eliminates the dependence of communication complexity on the total number of samples N in distributed learning, achieving complexity scaling only with the number of workers M, and outperforms Minibatch Accelerated SGD when M < O(N^{1/4}).
Communication overhead is a crucial bottleneck in scalable distributed learning. While existing methods aim to efficiently utilize data points, such as Local SGD, Minibatch SGD, and their accelerated variants, they still exhibit communication-round complexity that scales with the total number of samples $N$. In this paper, we introduce Local MixVR, a distributed framework that integrates local updates with variance-reduction techniques to mitigate local noise. We show that Local MixVR is the first distributed method to eliminate the dependence of communication complexity on $N$, achieving a complexity that scales only with the number of workers $M$. In common regimes where $M<O\left(N^{1/4}\right)$, Local MixVR outperforms the state-of-the-art Minibatch Accelerated SGD baseline, bridging a long-standing gap in distributed optimization and establishing a new paradigm for communication-efficient training.