Efficient and Private Federated Learning with Partially Trainable Networks
This addresses scalability and privacy issues for federated learning on resource-constrained mobile devices, though it is incremental as it builds on existing methods with a novel adaptation.
The paper tackles the challenge of communication and computation inefficiency in federated learning on edge devices by proposing partially trainable networks that freeze some parameters, achieving up to 46x reduction in communication cost with minimal accuracy loss.
Federated learning is used for decentralized training of machine learning models on a large number (millions) of edge mobile devices. It is challenging because mobile devices often have limited communication bandwidth and local computation resources. Therefore, improving the efficiency of federated learning is critical for scalability and usability. In this paper, we propose to leverage partially trainable neural networks, which freeze a portion of the model parameters during the entire training process, to reduce the communication cost with little implications on model performance. Through extensive experiments, we empirically show that Federated learning of Partially Trainable neural networks (FedPT) can result in superior communication-accuracy trade-offs, with up to $46\times$ reduction in communication cost, at a small accuracy cost. Our approach also enables faster training, with a smaller memory footprint, and better utility for strong differential privacy guarantees. The proposed FedPT method can be particularly interesting for pushing the limitations of over-parameterization in on-device learning.