Bias-Variance Reduced Local SGD for Less Heterogeneous Federated Learning
This work provides an incremental improvement in communication efficiency for federated learning practitioners dealing with nonconvex optimization and small data heterogeneity.
This paper introduces Bias-Variance Reduced Local SGD (BVR-L-SGD) to address the communication bottleneck in federated learning for nonconvex optimization. The method achieves better communication complexity than prior non-local and local methods, being the first to break the Θ(1/ε) barrier for general nonconvex smooth objectives under small heterogeneity and large local computation budgets.
Recently, local SGD has got much attention and been extensively studied in the distributed learning community to overcome the communication bottleneck problem. However, the superiority of local SGD to minibatch SGD only holds in quite limited situations. In this paper, we study a new local algorithm called Bias-Variance Reduced Local SGD (BVR-L-SGD) for nonconvex distributed optimization. Algorithmically, our proposed bias and variance reduced local gradient estimator fully utilizes small second-order heterogeneity of local objectives and suggests randomly picking up one of the local models instead of taking the average of them when workers are synchronized. Theoretically, under small heterogeneity of local objectives, we show that BVR-L-SGD achieves better communication complexity than both the previous non-local and local methods under mild conditions, and particularly BVR-L-SGD is the first method that breaks the barrier of communication complexity $Θ(1/\varepsilon)$ for general nonconvex smooth objectives when the heterogeneity is small and the local computation budget is large. Numerical results are given to verify the theoretical findings and give empirical evidence of the superiority of our method.