DFML: Decentralized Federated Mutual Learning
This addresses the problem of scalable and robust federated learning for real-world devices with heterogeneous data and models, though it appears incremental as it builds on mutual learning and decentralization concepts.
The paper tackled the challenges of communication bottlenecks, single points of failure, and model/data heterogeneity in Federated Learning by proposing a decentralized framework called DFML, which achieved substantial accuracy improvements, such as +17.20% and +19.95% on CIFAR-100 with 50 clients under IID and non-IID conditions.
In the realm of real-world devices, centralized servers in Federated Learning (FL) present challenges including communication bottlenecks and susceptibility to a single point of failure. Additionally, contemporary devices inherently exhibit model and data heterogeneity. Existing work lacks a Decentralized FL (DFL) framework capable of accommodating such heterogeneity without imposing architectural restrictions or assuming the availability of public data. To address these issues, we propose a Decentralized Federated Mutual Learning (DFML) framework that is serverless, supports nonrestrictive heterogeneous models, and avoids reliance on public data. DFML effectively handles model and data heterogeneity through mutual learning, which distills knowledge between clients, and cyclically varying the amount of supervision and distillation signals. Extensive experimental results demonstrate consistent effectiveness of DFML in both convergence speed and global accuracy, outperforming prevalent baselines under various conditions. For example, with the CIFAR-100 dataset and 50 clients, DFML achieves a substantial increase of +17.20% and +19.95% in global accuracy under Independent and Identically Distributed (IID) and non-IID data shifts, respectively.