Efficient and Reliable Overlay Networks for Decentralized Federated Learning
This work addresses scalability and robustness issues in decentralized federated learning for large-scale client networks, though it is incremental as it builds on existing graph theory and DFL frameworks.
The authors tackled the problem of slow convergence and poor generalization in decentralized federated learning by designing overlay networks based on d-regular expander graphs, resulting in accelerated convergence and improved generalization with theoretical guarantees.
We propose near-optimal overlay networks based on $d$-regular expander graphs to accelerate decentralized federated learning (DFL) and improve its generalization. In DFL a massive number of clients are connected by an overlay network, and they solve machine learning problems collaboratively without sharing raw data. Our overlay network design integrates spectral graph theory and the theoretical convergence and generalization bounds for DFL. As such, our proposed overlay networks accelerate convergence, improve generalization, and enhance robustness to clients failures in DFL with theoretical guarantees. Also, we present an efficient algorithm to convert a given graph to a practical overlay network and maintaining the network topology after potential client failures. We numerically verify the advantages of DFL with our proposed networks on various benchmark tasks, ranging from image classification to language modeling using hundreds of clients.