Communication Complexity of Distributed Convex Learning and Optimization
This work addresses communication bottlenecks in distributed machine learning, providing theoretical insights for researchers and practitioners, though it is incremental in refining existing understanding.
The paper investigates the fundamental communication limits for distributed convex learning and optimization, identifying conditions under which existing algorithms are optimal and revealing that many communication rounds are necessary when local objective functions lack similarity.
We study the fundamental limits to communication-efficient distributed methods for convex learning and optimization, under different assumptions on the information available to individual machines, and the types of functions considered. We identify cases where existing algorithms are already worst-case optimal, as well as cases where room for further improvement is still possible. Among other things, our results indicate that without similarity between the local objective functions (due to statistical data similarity or otherwise) many communication rounds may be required, even if the machines have unbounded computational power.