From PowerSGD to PowerSGD+: Low-Rank Gradient Compression for Distributed Optimization with Convergence Guarantees
This addresses convergence issues in communication-efficient distributed optimization for machine learning practitioners, though it is incremental as it builds on PowerSGD.
The paper tackled the problem of unclear convergence guarantees for PowerSGD in low-rank gradient compression for distributed optimization, showing it does not always converge and introducing PowerSGD+ with periodic subspace updates to ensure convergence, validated empirically on large language model tasks.
Low-rank gradient compression methods, such as PowerSGD, have gained attention in communication-efficient distributed optimization. However, the convergence guarantees of PowerSGD remain unclear, particularly in stochastic settings. In this paper, we show that PowerSGD does not always converge to the optimal solution and provide a clear counterexample to support this finding. To address this, we introduce PowerSGD+, which periodically updates the projection subspace via singular value decomposition, ensuring that it remains aligned with the optimal subspace. We prove that PowerSGD+ converges under standard assumptions and validate its effectiveness through empirical evaluation on large language model tasks.