TornadoAggregate: Accurate and Scalable Federated Learning via the Ring-Based Architecture
This work offers a significant improvement in accuracy and scalability for federated learning, particularly benefiting large-scale deployments with diverse client availability.
This paper addresses the high-variance problem in ring-based federated learning architectures, which are proposed to improve scalability and handle diurnal client properties. The authors reformulate loss minimization as a variance reduction problem and introduce three principles—Ring-Aware Grouping, Small Ring, and Ring Chaining—to reduce variance, resulting in up to 26.7% improved test accuracy and near-linear scalability.
Federated learning has emerged as a new paradigm of collaborative machine learning; however, many prior studies have used global aggregation along a star topology without much consideration of the communication scalability or the diurnal property relied on clients' local time variety. In contrast, ring architecture can resolve the scalability issue and even satisfy the diurnal property by iterating nodes without an aggregation. Nevertheless, such ring-based algorithms can inherently suffer from the high-variance problem. To this end, we propose a novel algorithm called TornadoAggregate that improves both accuracy and scalability by facilitating the ring architecture. In particular, to improve the accuracy, we reformulate the loss minimization into a variance reduction problem and establish three principles to reduce variance: Ring-Aware Grouping, Small Ring, and Ring Chaining. Experimental results show that TornadoAggregate improved the test accuracy by up to 26.7% and achieved near-linear scalability.