FedLion: Faster Adaptive Federated Optimization with Fewer Communication
This work addresses communication efficiency for distributed machine learning systems, but it is incremental as it adapts an existing centralized method to the federated setting.
The paper tackles the problem of slow convergence and high communication costs in Federated Learning by introducing FedLion, an adaptive optimization algorithm that outperforms state-of-the-art methods like FAFED and FedDA, reducing uplink data transmission and achieving faster convergence rates.
In Federated Learning (FL), a framework to train machine learning models across distributed data, well-known algorithms like FedAvg tend to have slow convergence rates, resulting in high communication costs during training. To address this challenge, we introduce FedLion, an adaptive federated optimization algorithm that seamlessly incorporates key elements from the recently proposed centralized adaptive algorithm, Lion (Chen et al. 2o23), into the FL framework. Through comprehensive evaluations on two widely adopted FL benchmarks, we demonstrate that FedLion outperforms previous state-of-the-art adaptive algorithms, including FAFED (Wu et al. 2023) and FedDA. Moreover, thanks to the use of signed gradients in local training, FedLion substantially reduces data transmission requirements during uplink communication when compared to existing adaptive algorithms, further reducing communication costs. Last but not least, this work also includes a novel theoretical analysis, showcasing that FedLion attains faster convergence rate than established FL algorithms like FedAvg.