FLECS-CGD: A Federated Learning Second-Order Framework via Compression and Sketching with Compressed Gradient Differences
This work addresses communication efficiency in federated learning, an incremental improvement over prior methods.
The paper tackles the communication bottleneck in federated learning by modifying the FLECS framework to compress gradient differences, enabling stochastic optimization and reducing costs. It provides convergence guarantees for strongly convex and nonconvex cases, with experiments showing practical benefits.
In the recent paper FLECS (Agafonov et al, FLECS: A Federated Learning Second-Order Framework via Compression and Sketching), the second-order framework FLECS was proposed for the Federated Learning problem. This method utilize compression of sketched Hessians to make communication costs low. However, the main bottleneck of FLECS is gradient communication without compression. In this paper, we propose the modification of FLECS with compressed gradient differences, which we call FLECS-CGD (FLECS with Compressed Gradient Differences) and make it applicable for stochastic optimization. Convergence guarantees are provided in strongly convex and nonconvex cases. Experiments show the practical benefit of proposed approach.