OpenDiLoCo: An Open-Source Framework for Globally Distributed Low-Communication Training
This provides a practical solution for researchers and practitioners needing efficient, globally distributed training of large models, though it's primarily an implementation and extension of existing work.
The researchers tackled the challenge of distributed training for large language models by developing OpenDiLoCo, an open-source framework that replicates and extends the DiLoCo method, achieving 90-95% compute utilization across geographically distributed workers and scaling to billion-parameter models.
OpenDiLoCo is an open-source implementation and replication of the Distributed Low-Communication (DiLoCo) training method for large language models. We provide a reproducible implementation of the DiLoCo experiments, offering it within a scalable, decentralized training framework using the Hivemind library. We demonstrate its effectiveness by training a model across two continents and three countries, while maintaining 90-95% compute utilization. Additionally, we conduct ablations studies focusing on the algorithm's compute efficiency, scalability in the number of workers and show that its gradients can be all-reduced using FP16 without any performance degradation. Furthermore, we scale OpenDiLoCo to 3x the size of the original work, demonstrating its effectiveness for billion parameter models.