Decentralised Federated Learning over Temporal Networks: The Role of Heterogeneities
For researchers in federated learning, this work reveals that common experimental setups overestimate convergence speed by neglecting network heterogeneities, highlighting a critical gap in realistic evaluation.
The paper investigates how structural and temporal inhomogeneities in peer-to-peer communication networks affect decentralised federated learning, showing that ignoring these heterogeneities leads to unrealistically rapid convergence in typical experiments, and that real-world inhomogeneities dramatically slow down convergence.
Decentralised federated learning, based on peer-to-peer communication, is increasingly proposed for on-device training of machine learning models, promising a privacy-preserving, communication-efficient training process with no risk of single-point failure. However, the role of structural and temporal inhomogeneities in such fully decentralised settings remains poorly understood. Here, we investigate their effects when model parameters are locally averaged during aggregation. We show that the decentralised federated learning process is governed, both in the early phase and the late, stationary limit, by the same dynamics as a lazy random-walk diffusion process on temporal networks. Based on this mapping, we demonstrate that the typical experimental scenario used in decentralised federated learning leads to unrealistically rapid convergence because of ignoring the temporal and structural inhomogeneities inherent in the communication network. We analyse real-world temporal networks and find that inhomogeneities most often dramatically slow down diffusion, hence the convergence process.