Multi-Level Branched Regularization for Federated Learning
This addresses data inconsistency issues in federated learning for distributed systems, representing an incremental improvement with novel architectural regularization.
The paper tackles data heterogeneity and imbalance in federated learning by proposing a multi-level branched regularization technique that grafts local and global subnetworks, achieving remarkable performance gains in accuracy and efficiency compared to existing methods.
A critical challenge of federated learning is data heterogeneity and imbalance across clients, which leads to inconsistency between local networks and unstable convergence of global models. To alleviate the limitations, we propose a novel architectural regularization technique that constructs multiple auxiliary branches in each local model by grafting local and global subnetworks at several different levels and that learns the representations of the main pathway in the local model congruent to the auxiliary hybrid pathways via online knowledge distillation. The proposed technique is effective to robustify the global model even in the non-iid setting and is applicable to various federated learning frameworks conveniently without incurring extra communication costs. We perform comprehensive empirical studies and demonstrate remarkable performance gains in terms of accuracy and efficiency compared to existing methods. The source code is available at our project page.