SignSGD with Federated Defense: Harnessing Adversarial Attacks through Gradient Sign Decoding
This addresses communication efficiency and robustness in distributed learning for machine learning practitioners, but is incremental as it builds on existing SignSGD methods.
The paper tackles the problem of adversarial workers degrading convergence in distributed learning with SignSGD, showing that convergence rate remains invariant as adversarial workers increase if they are outnumbered by benign workers, and introduces signSGD-FD which achieves superior convergence rates in experiments.
Distributed learning is an effective approach to accelerate model training using multiple workers. However, substantial communication delays emerge between workers and a parameter server due to massive costs associated with communicating gradients. SignSGD with majority voting (signSGD-MV) is a simple yet effective optimizer that reduces communication costs through one-bit quantization, yet the convergence rates considerably decrease as adversarial workers increase. In this paper, we show that the convergence rate is invariant as the number of adversarial workers increases, provided that the number of adversarial workers is smaller than that of benign workers. The key idea showing this counter-intuitive result is our novel signSGD with federated defense (signSGD-FD). Unlike the traditional approaches, signSGD-FD exploits the gradient information sent by adversarial workers with the proper weights, which are obtained through gradient sign decoding. Experimental results demonstrate signSGD-FD achieves superior convergence rates over traditional algorithms in various adversarial attack scenarios.