An Efficient Subspace Algorithm for Federated Learning on Heterogeneous Data
It addresses efficiency and drift issues in federated learning for large-scale deep neural networks, which is incremental as it builds on existing methods with subspace techniques.
This work tackled the challenges of federated learning on heterogeneous data, such as client drift and high costs, by proposing FedSub, an efficient subspace algorithm that reduces communication, computation, and memory costs and mitigates drift, with experimental results showing its efficiency.
This work addresses the key challenges of applying federated learning to large-scale deep neural networks, particularly the issue of client drift due to data heterogeneity across clients and the high costs of communication, computation, and memory. We propose FedSub, an efficient subspace algorithm for federated learning on heterogeneous data. Specifically, FedSub utilizes subspace projection to guarantee local updates of each client within low-dimensional subspaces, thereby reducing communication, computation, and memory costs. Additionally, it incorporates low-dimensional dual variables to mitigate client drift. We provide convergence analysis that reveals the impact of key factors such as step size and subspace projection matrices on convergence. Experimental results demonstrate its efficiency.