LoSAC: An Efficient Local Stochastic Average Control Method for Federated Optimization
This work addresses communication bottlenecks and security risks in federated learning for distributed clients, representing an incremental improvement over existing methods.
The paper tackles the problems of model divergence and communication inefficiency in federated optimization by proposing LoSAC, a method that locally updates the global gradient estimate after each local model update, resulting in over 100% average improvement in communication efficiency and enhanced defense against information leakage.
Federated optimization (FedOpt), which targets at collaboratively training a learning model across a large number of distributed clients, is vital for federated learning. The primary concerns in FedOpt can be attributed to the model divergence and communication efficiency, which significantly affect the performance. In this paper, we propose a new method, i.e., LoSAC, to learn from heterogeneous distributed data more efficiently. Its key algorithmic insight is to locally update the estimate for the global full gradient after {each} regular local model update. Thus, LoSAC can keep clients' information refreshed in a more compact way. In particular, we have studied the convergence result for LoSAC. Besides, the bonus of LoSAC is the ability to defend the information leakage from the recent technique Deep Leakage Gradients (DLG). Finally, experiments have verified the superiority of LoSAC comparing with state-of-the-art FedOpt algorithms. Specifically, LoSAC significantly improves communication efficiency by more than $100\%$ on average, mitigates the model divergence problem and equips with the defense ability against DLG.