Efficient Language Model Architectures for Differentially Private Federated Learning
This work addresses the challenge of training language models on distributed edge devices without data leaving them, offering incremental improvements for federated learning systems.
The paper tackled the problem of efficiently training language models in cross-device federated learning by proposing a scale-invariant modification to recurrent networks, which achieved faster convergence and better utility than standard models in large-scale experiments, with improvements also shown for transformers and differential privacy settings.
Cross-device federated learning (FL) is a technique that trains a model on data distributed across typically millions of edge devices without data leaving the devices. SGD is the standard client optimizer for on device training in cross-device FL, favored for its memory and computational efficiency. However, in centralized training of neural language models, adaptive optimizers are preferred as they offer improved stability and performance. In light of this, we ask if language models can be modified such that they can be efficiently trained with SGD client optimizers and answer this affirmatively. We propose a scale-invariant Coupled Input Forget Gate (SI CIFG) recurrent network by modifying the sigmoid and tanh activations in the recurrent cell and show that this new model converges faster and achieves better utility than the standard CIFG recurrent model in cross-device FL in large scale experiments. We further show that the proposed scale invariant modification also helps in federated learning of larger transformer models. Finally, we demonstrate the scale invariant modification is also compatible with other non-adaptive algorithms. Particularly, our results suggest an improved privacy utility trade-off in federated learning with differential privacy.