On the Communication Latency of Wireless Decentralized Learning
This addresses communication bottlenecks in decentralized machine learning for wireless networks, providing theoretical insights but is incremental as it builds on existing models.
The paper tackles the problem of communication latency in wireless decentralized learning by analyzing a network of nodes optimizing a global objective function, showing that the communication delay per round scales as O(n^{2-3β}/(β log n)), increasing with node count and gradient exchange distance.
We consider a wireless network comprising $n$ nodes located within a circular area of radius $R$, which are participating in a decentralized learning algorithm to optimize a global objective function using their local datasets. To enable gradient exchanges across the network, we assume each node communicates only with a set of neighboring nodes, which are within a distance $R n^{-β}$ of itself, where $β\in(0,\frac{1}{2})$. We use tools from network information theory and random geometric graph theory to show that the communication delay for a single round of exchanging gradients on all the links throughout the network scales as $\mathcal{O}\left(\frac{n^{2-3β}}{β\log n}\right)$, increasing (at different rates) with both the number of nodes and the gradient exchange threshold distance.