Stochastic Alternating Direction Method of Multipliers for Byzantine-Robust Distributed Learning
This addresses the problem of securing distributed learning systems against malicious workers for applications like federated learning, though it is incremental as it builds on prior TV norm-penalized formulations.
The paper tackles distributed learning under Byzantine attacks by proposing a stochastic ADMM method that converges to a bounded neighborhood of the optimal solution at a rate of O(1/k), with experiments on MNIST and COVERTYPE datasets showing effectiveness against various attacks.
This paper aims to solve a distributed learning problem under Byzantine attacks. In the underlying distributed system, a number of unknown but malicious workers (termed as Byzantine workers) can send arbitrary messages to the master and bias the learning process, due to data corruptions, computation errors or malicious attacks. Prior work has considered a total variation (TV) norm-penalized approximation formulation to handle the Byzantine attacks, where the TV norm penalty forces the regular workers' local variables to be close, and meanwhile, tolerates the outliers sent by the Byzantine workers. To solve the TV norm-penalized approximation formulation, we propose a Byzantine-robust stochastic alternating direction method of multipliers (ADMM) that fully utilizes the separable problem structure. Theoretically, we prove that the proposed method converges to a bounded neighborhood of the optimal solution at a rate of O(1/k) under mild assumptions, where k is the number of iterations and the size of neighborhood is determined by the number of Byzantine workers. Numerical experiments on the MNIST and COVERTYPE datasets demonstrate the effectiveness of the proposed method to various Byzantine attacks.