Minibatch vs Local SGD for Heterogeneous Distributed Learning
This work addresses optimization challenges in distributed settings like federated learning, where data heterogeneity across machines complicates training, offering incremental theoretical improvements.
The paper analyzes Local SGD and Minibatch SGD for heterogeneous distributed learning, showing that Minibatch SGD outperforms existing Local SGD analyses, accelerated Minibatch SGD is optimal under high heterogeneity, and provides the first Local SGD upper bound that beats Minibatch SGD in non-homogeneous cases.
We analyze Local SGD (aka parallel or federated SGD) and Minibatch SGD in the heterogeneous distributed setting, where each machine has access to stochastic gradient estimates for a different, machine-specific, convex objective; the goal is to optimize w.r.t. the average objective; and machines can only communicate intermittently. We argue that, (i) Minibatch SGD (even without acceleration) dominates all existing analysis of Local SGD in this setting, (ii) accelerated Minibatch SGD is optimal when the heterogeneity is high, and (iii) present the first upper bound for Local SGD that improves over Minibatch SGD in a non-homogeneous regime.