Communication Compression for Decentralized Learning with Operator Splitting Methods
This addresses communication bottlenecks for decentralized learning systems with heterogeneous data, offering an incremental improvement over existing methods.
The paper tackles the high communication costs in decentralized learning with operator splitting methods like Edge-Consensus Learning (ECL) by proposing a compression framework called C-ECL, which achieves nearly equivalent performance with fewer parameter exchanges and is more robust to heterogeneous data than Gossip-based algorithms.
In decentralized learning, operator splitting methods using a primal-dual formulation (e.g., the Edge-Consensus Learning (ECL)) has been shown to be robust to heterogeneous data and has attracted significant attention in recent years. However, in the ECL, a node needs to exchange dual variables with its neighbors. These exchanges incur significant communication costs. For the Gossip-based algorithms, many compression methods have been proposed, but these Gossip-based algorithm do not perform well when the data distribution held by each node is statistically heterogeneous. In this work, we propose the novel framework of the compression methods for the ECL, called the Communication Compressed ECL (C-ECL). Specifically, we reformulate the update formulas of the ECL, and propose to compress the update values of the dual variables. We demonstrate experimentally that the C-ECL can achieve a nearly equivalent performance with fewer parameter exchanges than the ECL. Moreover, we demonstrate that the C-ECL is more robust to heterogeneous data than the Gossip-based algorithms.