Byzantine Resilient Distributed Multi-Task Learning
This addresses the vulnerability of distributed multi-task learning in multi-agent networks to Byzantine failures, offering a solution for scenarios with heterogeneous data sources, though it is incremental as it builds on existing distributed learning frameworks.
The paper tackles the problem of Byzantine agents disrupting distributed multi-task learning by proposing a resilient approach with an online weight assignment rule based on accumulated loss and a filtering step, showing that normal agents converge to the global minimum for convex models and achieve improved expected regret, with empirical validation on regression, classification, and non-convex models like CNNs.
Distributed multi-task learning provides significant advantages in multi-agent networks with heterogeneous data sources where agents aim to learn distinct but correlated models simultaneously.However, distributed algorithms for learning relatedness among tasks are not resilient in the presence of Byzantine agents. In this paper, we present an approach for Byzantine resilient distributed multi-task learning. We propose an efficient online weight assignment rule by measuring the accumulated loss using an agent's data and its neighbors' models. A small accumulated loss indicates a large similarity between the two tasks. In order to ensure the Byzantine resilience of the aggregation at a normal agent, we introduce a step for filtering out larger losses. We analyze the approach for convex models and show that normal agents converge resiliently towards the global minimum.Further, aggregation with the proposed weight assignment rule always results in an improved expected regret than the non-cooperative case. Finally, we demonstrate the approach using three case studies, including regression and classification problems, and show that our method exhibits good empirical performance for non-convex models, such as convolutional neural networks.