Decentralized Stochastic Subgradient-type Methods with Communication Compression for Nonsmooth Nonconvex Optimization
It provides a theoretical foundation for decentralized optimization with compressed communication in challenging nonsmooth nonconvex settings, which is relevant for large-scale distributed machine learning.
This paper proposes a unified framework for decentralized stochastic subgradient-type methods with communication compression, establishing global convergence for nonsmooth nonconvex optimization without Clarke regularity. Numerical experiments validate the theoretical results and demonstrate communication-accuracy trade-offs.
In this paper, we consider the nonsmooth nonconvex decentralized optimization problem, where inter-agent communication is compressed. We propose a general framework that unifies various decentralized stochastic subgradient-type methods with unbiased compression and contractive compression with error compensation. By relating the consensus-error iterates and the averaged iterates to the trajectories of continuous-time differential inclusions, we establish global convergence for all methods encompassed by our framework when the objective functions are nonsmooth and lack Clarke regularity. Based on our framework, we further develop several compression-based methods, including decentralized stochastic subgradient methods utilizing sign-based regularization and gradient-tracking momentum. Preliminary numerical experiments empirically support our theoretical results and highlight the communication-accuracy trade-off of the newly developed methods.