DePRL: Achieving Linear Convergence Speedup in Personalized Decentralized Learning with Shared Representations
This addresses the issue of poor model performance on individual workers in decentralized learning due to data heterogeneity, offering a scalable solution for distributed environments.
The authors tackled the problem of data heterogeneity in decentralized learning by proposing DePRL, a personalized algorithm that learns a shared global representation and local heads, achieving a provable linear speedup in convergence with respect to the number of workers.
Decentralized learning has emerged as an alternative method to the popular parameter-server framework which suffers from high communication burden, single-point failure and scalability issues due to the need of a central server. However, most existing works focus on a single shared model for all workers regardless of the data heterogeneity problem, rendering the resulting model performing poorly on individual workers. In this work, we propose a novel personalized decentralized learning algorithm named DePRL via shared representations. Our algorithm relies on ideas from representation learning theory to learn a low-dimensional global representation collaboratively among all workers in a fully decentralized manner, and a user-specific low-dimensional local head leading to a personalized solution for each worker. We show that DePRL achieves, for the first time, a provable linear speedup for convergence with general non-linear representations (i.e., the convergence rate is improved linearly with respect to the number of workers). Experimental results support our theoretical findings showing the superiority of our method in data heterogeneous environments.