Trustworthy Efficient Communication for Distributed Learning using LQ-SGD Algorithm
This addresses communication bottlenecks for distributed learning systems, with incremental improvements over existing compression methods like PowerSGD.
The paper tackles the problem of high communication overhead in distributed training by proposing LQ-SGD, a gradient compression algorithm that combines low-rank approximation and log-quantization techniques, achieving drastic reduction in communication while maintaining convergence speed and model accuracy.
We propose LQ-SGD (Low-Rank Quantized Stochastic Gradient Descent), an efficient communication gradient compression algorithm designed for distributed training. LQ-SGD further develops on the basis of PowerSGD by incorporating the low-rank approximation and log-quantization techniques, which drastically reduce the communication overhead, while still ensuring the convergence speed of training and model accuracy. In addition, LQ-SGD and other compression-based methods show stronger resistance to gradient inversion than traditional SGD, providing a more robust and efficient optimization path for distributed learning systems.