Byzantine-Robust Learning on Heterogeneous Datasets via Bucketing
This addresses a critical vulnerability in federated learning for real-world applications with non-iid data, representing a foundational advance rather than an incremental improvement.
The paper tackles the problem of Byzantine-robust distributed learning on heterogeneous (non-iid) datasets, where existing defenses fail under new attacks, and proposes a bucketing scheme that adapts robust algorithms to achieve guaranteed convergence with negligible computational cost.
In Byzantine robust distributed or federated learning, a central server wants to train a machine learning model over data distributed across multiple workers. However, a fraction of these workers may deviate from the prescribed algorithm and send arbitrary messages. While this problem has received significant attention recently, most current defenses assume that the workers have identical data. For realistic cases when the data across workers are heterogeneous (non-iid), we design new attacks which circumvent current defenses, leading to significant loss of performance. We then propose a simple bucketing scheme that adapts existing robust algorithms to heterogeneous datasets at a negligible computational cost. We also theoretically and experimentally validate our approach, showing that combining bucketing with existing robust algorithms is effective against challenging attacks. Our work is the first to establish guaranteed convergence for the non-iid Byzantine robust problem under realistic assumptions.