Federated Word2Vec: Leveraging Federated Learning to Encourage Collaborative Representation Learning
This addresses privacy and regulatory issues for organizations needing collaborative NLP training, but it is incremental as it adapts an existing method to a new context.
The paper tackled the challenge of training NLP models like Word2Vec without centralizing data due to privacy concerns, by applying Federated Learning across organizations with large corpora, and found that it maintained result quality and convergence time compared to centralized training.
Large scale contextual representation models have significantly advanced NLP in recent years, understanding the semantics of text to a degree never seen before. However, they need to process large amounts of data to achieve high-quality results. Joining and accessing all these data from multiple sources can be extremely challenging due to privacy and regulatory reasons. Federated Learning can solve these limitations by training models in a distributed fashion, taking advantage of the hardware of the devices that generate the data. We show the viability of training NLP models, specifically Word2Vec, with the Federated Learning protocol. In particular, we focus on a scenario in which a small number of organizations each hold a relatively large corpus. The results show that neither the quality of the results nor the convergence time in Federated Word2Vec deteriorates as compared to centralised Word2Vec.