Decentralized Federated Learning by Partial Message Exchange
This work addresses communication and privacy issues in decentralized federated learning for large-scale heterogeneous networks, representing an incremental improvement over existing methods.
The paper tackles the challenges of decentralized federated learning, such as data heterogeneity and high communication costs, by proposing PaME, an algorithm that exchanges only randomly selected sparse coordinates between nodes, achieving substantial reductions in communication costs while preserving privacy and accuracy, with numerical experiments showing superior performance compared to existing methods.
Decentralized federated learning (DFL) has emerged as a transformative server-free paradigm that enables collaborative learning over large-scale heterogeneous networks. However, it continues to face fundamental challenges, including data heterogeneity, restrictive assumptions for theoretical analysis, and degraded convergence when standard communication- or privacyenhancing techniques are applied. To overcome these drawbacks, this paper develops a novel algorithm, PaME (DFL by Partial Message Exchange). The central principle is to allow only randomly selected sparse coordinates to be exchanged between two neighbor nodes. Consequently, PaME achieves substantial reductions in communication costs while still preserving a high level of privacy, without sacrificing accuracy. Moreover, grounded in rigorous analysis, the algorithm is shown to converge at a linear rate under the gradient to be locally Lipschitz continuous and the communication matrix to be doubly stochastic. These two mild assumptions not only dispense with many restrictive conditions commonly imposed by existing DFL methods but also enables PaME to effectively address data heterogeneity. Furthermore, comprehensive numerical experiments demonstrate its superior performance compared with several representative decentralized learning algorithms.