Semi-Decentralized Federated Learning with Collaborative Relaying
This work addresses connectivity issues in federated learning for distributed systems, but it is incremental as it builds on existing federated averaging methods.
The paper tackles the problem of intermittent connectivity in federated learning by proposing a semi-decentralized algorithm where clients relay neighbors' updates to a central server, optimizing weights to reduce variance and improve convergence. Numerical simulations show improved convergence rate and accuracy compared to federated averaging in settings with intermittent connectivity.
We present a semi-decentralized federated learning algorithm wherein clients collaborate by relaying their neighbors' local updates to a central parameter server (PS). At every communication round to the PS, each client computes a local consensus of the updates from its neighboring clients and eventually transmits a weighted average of its own update and those of its neighbors to the PS. We appropriately optimize these averaging weights to ensure that the global update at the PS is unbiased and to reduce the variance of the global update at the PS, consequently improving the rate of convergence. Numerical simulations substantiate our theoretical claims and demonstrate settings with intermittent connectivity between the clients and the PS, where our proposed algorithm shows an improved convergence rate and accuracy in comparison with the federated averaging algorithm.