Decentralized Learning over Wireless Networks with Broadcast-Based Subgraph Sampling
This work addresses communication efficiency for decentralized learning in wireless networks, representing an incremental improvement over prior scheduling approaches.
The paper tackles the communication bottleneck in decentralized learning over wireless networks by proposing BASS, a broadcast-based subgraph sampling framework for D-SGD, which achieves faster convergence per transmission slot compared to existing link-based methods.
This work centers on the communication aspects of decentralized learning over wireless networks, using consensus-based decentralized stochastic gradient descent (D-SGD). Considering the actual communication cost or delay caused by in-network information exchange in an iterative process, our goal is to achieve fast convergence of the algorithm measured by improvement per transmission slot. We propose BASS, an efficient communication framework for D-SGD over wireless networks with broadcast transmission and probabilistic subgraph sampling. In each iteration, we activate multiple subsets of non-interfering nodes to broadcast model updates to their neighbors. These subsets are randomly activated over time, with probabilities reflecting their importance in network connectivity and subject to a communication cost constraint (e.g., the average number of transmission slots per iteration). During the consensus update step, only bi-directional links are effectively preserved to maintain communication symmetry. In comparison to existing link-based scheduling methods, the inherent broadcasting nature of wireless channels offers intrinsic advantages in speeding up convergence of decentralized learning by creating more communicated links with the same number of transmission slots.