Get More for Less in Decentralized Learning Systems
This addresses the problem of network overload for users of decentralized learning systems, offering a communication-efficient solution that is incremental over existing methods.
The paper tackles the challenge of high communication overhead in decentralized learning systems by proposing JWINS, a method that shares only a subset of parameters through sparsification, achieving similar accuracies to full-sharing while sending up to 64% fewer bytes and outperforming CHOCO-SGD by up to 4x in network savings and time on low budgets.
Decentralized learning (DL) systems have been gaining popularity because they avoid raw data sharing by communicating only model parameters, hence preserving data confidentiality. However, the large size of deep neural networks poses a significant challenge for decentralized training, since each node needs to exchange gigabytes of data, overloading the network. In this paper, we address this challenge with JWINS, a communication-efficient and fully decentralized learning system that shares only a subset of parameters through sparsification. JWINS uses wavelet transform to limit the information loss due to sparsification and a randomized communication cut-off that reduces communication usage without damaging the performance of trained models. We demonstrate empirically with 96 DL nodes on non-IID datasets that JWINS can achieve similar accuracies to full-sharing DL while sending up to 64% fewer bytes. Additionally, on low communication budgets, JWINS outperforms the state-of-the-art communication-efficient DL algorithm CHOCO-SGD by up to 4x in terms of network savings and time.