Byzantine-Robust Decentralized Learning via ClippedGossip
It addresses robust distributed learning for decentralized systems vulnerable to adversarial attacks, representing a novel method for a known bottleneck.
The paper tackles Byzantine-robust decentralized training on arbitrary graphs by proposing ClippedGossip, which provably converges to a neighborhood of the stationary point for non-convex objectives, with empirical performance demonstrated under attacks.
In this paper, we study the challenging task of Byzantine-robust decentralized training on arbitrary communication graphs. Unlike federated learning where workers communicate through a server, workers in the decentralized environment can only talk to their neighbors, making it harder to reach consensus and benefit from collaborative training. To address these issues, we propose a ClippedGossip algorithm for Byzantine-robust consensus and optimization, which is the first to provably converge to a $O(δ_{\max}ζ^2/γ^2)$ neighborhood of the stationary point for non-convex objectives under standard assumptions. Finally, we demonstrate the encouraging empirical performance of ClippedGossip under a large number of attacks.