Personalized Decentralized Multi-Task Learning Over Dynamic Communication Graphs
This work addresses data heterogeneity for users in decentralized learning settings, offering an incremental improvement over existing personalized methods.
The paper tackles the challenge of data heterogeneity in decentralized multi-task learning by proposing an algorithm that dynamically adjusts communication graphs based on task correlations, leading to faster convergence and improved performance compared to fully-connected networks, as demonstrated on synthetic and CelebA datasets.
Decentralized and federated learning algorithms face data heterogeneity as one of the biggest challenges, especially when users want to learn a specific task. Even when personalized headers are used concatenated to a shared network (PF-MTL), aggregating all the networks with a decentralized algorithm can result in performance degradation as a result of heterogeneity in the data. Our algorithm uses exchanged gradients to calculate the correlations among tasks automatically, and dynamically adjusts the communication graph to connect mutually beneficial tasks and isolate those that may negatively impact each other. This algorithm improves the learning performance and leads to faster convergence compared to the case where all clients are connected to each other regardless of their correlations. We conduct experiments on a synthetic Gaussian dataset and a large-scale celebrity attributes (CelebA) dataset. The experiment with the synthetic data illustrates that our proposed method is capable of detecting tasks that are positively and negatively correlated. Moreover, the results of the experiments with CelebA demonstrate that the proposed method may produce significantly faster training results than fully-connected networks.