Factored Gossip DiLoCo: Reducing Blocking Communication in DiLoCo
For practitioners of large-scale distributed training in low-bandwidth or unreliable networks, this work offers a practical way to reduce synchronization overhead and improve fault tolerance.
This paper introduces Factored Gossip DiLoCo, a method that reduces blocking communication in distributed training by replacing exact synchronization with approximate gossip-based mixing, improving compute utilization and robustness to failures in low-bandwidth settings while maintaining comparable training progress to DiLoCo.
To make large-scale distributed training practical outside high-bandwidth datacenters, we must reduce blocking, high-volume synchronization. While DiLoCo communicates infrequently, its outer synchronization remains bandwidth-heavy and brittle to stragglers and transient failures. We relax exact synchronization to approximate synchronization via mixing/gossip, which degrades gracefully under delays and communication failures. This allows us to factorize DiLoCo synchronization into a non-blocking mixing step that overlaps computation with no staleness, and a blocking mixing step that tightens worker agreement, yielding a tunable trade-off between compute utilization and optimization stability. On up to billion-parameter language models in low-bandwidth settings, our framework substantially improves compute utilization compared to DiLoCo, with training progress ranging from comparable to closely matching it, and is more robust to failures.