LoRDO: Distributed Low-Rank Optimization with Infrequent Communication
This work addresses communication and memory constraints in distributed training of foundation models, offering a practical improvement for large-scale AI deployments.
The paper tackles the problem of distributed training bottlenecks from optimizer state memory and communication by proposing LoRDO, a framework that unifies low-rank optimization with infrequent synchronization. It achieves near-parity performance with low-rank DDP on language modeling tasks at model scales up to 720M parameters while reducing communication by approximately 10 times.
Distributed training of foundation models via $\texttt{DDP}$ is limited by interconnect bandwidth. While infrequent communication strategies reduce synchronization frequency, they remain bottlenecked by the memory and communication requirements of optimizer states. Low-rank optimizers can alleviate these constraints; however, in the local-update regime, workers lack access to the full-batch gradients required to compute low-rank projections, which degrades performance. We propose $\texttt{LoRDO}$, a principled framework unifying low-rank optimization with infrequent synchronization. We first demonstrate that, while global projections based on pseudo-gradients are theoretically superior, they permanently restrict the optimization trajectory to a low-rank subspace. To restore subspace exploration, we introduce a full-rank quasi-hyperbolic update. $\texttt{LoRDO}$ achieves near-parity with low-rank $\texttt{DDP}$ in language modeling and downstream tasks at model scales of $125$M--$720$M, while reducing communication by $\approx 10 \times$. Finally, we show that $\texttt{LoRDO}$ improves performance even more in very low-memory settings with small rank/batch size.