Byzantine-Resilient Zero-Order Optimization for Communication-Efficient Heterogeneous Federated Learning
This addresses communication efficiency and security for federated learning systems, particularly in heterogeneous and adversarial environments, representing a novel method for a known bottleneck.
The paper tackles the problem of Byzantine attacks and high communication costs in federated learning by introducing CyBeR-0, a zero-order optimization method that achieves stable performance with only a few scalars per-round communication and reduced memory requirements, as shown in empirical evaluations on standard tasks and fine-tuning large language models.
We introduce CyBeR-0, a Byzantine-resilient federated zero-order optimization method that is robust under Byzantine attacks and provides significant savings in uplink and downlink communication costs. We introduce transformed robust aggregation to give convergence guarantees for general non-convex objectives under client data heterogeneity. Empirical evaluations for standard learning tasks and fine-tuning large language models show that CyBeR-0 exhibits stable performance with only a few scalars per-round communication cost and reduced memory requirements.