Communication-Efficient Byzantine-Resilient Federated Zero-Order Optimization
This addresses the challenge of secure and efficient federated learning for distributed systems, though it appears incremental as it builds on existing zero-order and Byzantine-resilient approaches.
The paper tackles the problem of communication and memory efficiency in federated learning with Byzantine faults by introducing CYBER-0, a zero-order optimization algorithm, and shows it outperforms state-of-the-art methods in efficiency while achieving similar accuracy on MNIST and RoBERTa-Large tasks.
We introduce CYBER-0, the first zero-order optimization algorithm for memory-and-communication efficient Federated Learning, resilient to Byzantine faults. We show through extensive numerical experiments on the MNIST dataset and finetuning RoBERTa-Large that CYBER-0 outperforms state-of-the-art algorithms in terms of communication and memory efficiency while reaching similar accuracy. We provide theoretical guarantees on its convergence for convex loss functions.