Pretrained Models for Multilingual Federated Learning
This addresses the challenge of non-IID multilingual data in federated learning for NLP, which is incremental as it builds on existing FL methods.
The paper tackled the problem of multilingual text in federated learning (FL) for NLP tasks, showing that using pretrained models reduces FL's negative effects, allowing performance near or better than centralized learning even with non-IID data.
Since the advent of Federated Learning (FL), research has applied these methods to natural language processing (NLP) tasks. Despite a plethora of papers in FL for NLP, no previous works have studied how multilingual text impacts FL algorithms. Furthermore, multilingual text provides an interesting avenue to examine the impact of non-IID text (e.g. different languages) on FL in naturally occurring data. We explore three multilingual language tasks, language modeling, machine translation, and text classification using differing federated and non-federated learning algorithms. Our results show that using pretrained models reduces the negative effects of FL, helping them to perform near or better than centralized (no privacy) learning, even when using non-IID partitioning.