Local Stochastic Gradient Descent Ascent: Convergence Analysis and Communication Efficiency
This work addresses communication efficiency for large-scale minimax learning problems, offering a novel method that is incremental but extends local SGD to minimax optimization.
The paper tackles the communication overhead in distributed minimax optimization, such as in adversarial robust learning and GANs, by proposing local Stochastic Gradient Descent Ascent (local SGDA), which reduces synchronization frequency and provably converges with established rates in various settings, including a novel variant for nonconvex-nonconcave problems.
Local SGD is a promising approach to overcome the communication overhead in distributed learning by reducing the synchronization frequency among worker nodes. Despite the recent theoretical advances of local SGD in empirical risk minimization, the efficiency of its counterpart in minimax optimization remains unexplored. Motivated by large scale minimax learning problems, such as adversarial robust learning and training generative adversarial networks (GANs), we propose local Stochastic Gradient Descent Ascent (local SGDA), where the primal and dual variables can be trained locally and averaged periodically to significantly reduce the number of communications. We show that local SGDA can provably optimize distributed minimax problems in both homogeneous and heterogeneous data with reduced number of communications and establish convergence rates under strongly-convex-strongly-concave and nonconvex-strongly-concave settings. In addition, we propose a novel variant local SGDA+, to solve nonconvex-nonconcave problems. We give corroborating empirical evidence on different distributed minimax problems.