Accelerated Methods with Compressed Communications for Distributed Optimization Problems under Data Similarity
This work addresses the communication challenge in distributed learning, particularly for high-dimensional data, by introducing a novel synergy of methods, though it appears incremental in building on existing compression and local step techniques.
The paper tackles the communication bottleneck in distributed optimization by proposing accelerated algorithms that combine compression techniques with data similarity, achieving record results as confirmed by experiments on various datasets and average losses.
In recent years, as data and problem sizes have increased, distributed learning has become an essential tool for training high-performance models. However, the communication bottleneck, especially for high-dimensional data, is a challenge. Several techniques have been developed to overcome this problem. These include communication compression and implementation of local steps, which work particularly well when there is similarity of local data samples. In this paper, we study the synergy of these approaches for efficient distributed optimization. We propose the first theoretically grounded accelerated algorithms utilizing unbiased and biased compression under data similarity, leveraging variance reduction and error feedback frameworks. Our results are of record and confirmed by experiments on different average losses and datasets.