ColNet: Collaborative Optimization in Decentralized Federated Multi-task Learning Systems
This addresses task heterogeneity in decentralized federated learning, which is an incremental improvement over prior work focused on data heterogeneity.
The paper tackles the problem of task heterogeneity in decentralized federated multi-task learning by proposing ColNet, which uses adaptive clustering and cross-group aggregation to outperform existing methods in terms of performance and robustness to attacks.
The integration of Federated Learning (FL) and Multi-Task Learning (MTL) has been explored to address client heterogeneity, with Federated Multi-Task Learning (FMTL) treating each client as a distinct task. However, most existing research focuses on data heterogeneity (e.g., addressing non-IID data) rather than task heterogeneity, where clients solve fundamentally different tasks. Additionally, much of the work relies on centralized settings with a server managing the federation, leaving the more challenging domain of decentralized FMTL largely unexplored. Thus, this work bridges this gap by proposing ColNet, a framework designed for heterogeneous tasks in decentralized federated environments. ColNet partitions models into a backbone and task-specific heads, and uses adaptive clustering based on model and data sensitivity to form task-coherent client groups. Backbones are averaged within groups, and group leaders perform hyper-conflict-averse cross-group aggregation. Across datasets and federations, ColNet outperforms competing schemes under label and task heterogeneity and shows robustness to poisoning attacks.