AdLoCo: adaptive batching significantly improves communications efficiency and convergence for Large Language Models
This work addresses the challenge of scaling LLM training for researchers and engineers by enhancing communications efficiency and convergence, though it is incremental as it builds on existing methods like DiLoCo.
The paper tackles the problem of inefficient distributed training of Large Language Models under dynamic workloads by proposing a three-stage method that combines Multi-Instance Training, Adaptive Batched DiLoCo, and a switch mode mechanism, resulting in improved convergence speed and system efficiency with a theoretical estimate of communication requirements.
Scaling distributed training of Large Language Models (LLMs) requires not only algorithmic advances but also efficient utilization of heterogeneous hardware resources. While existing methods such as DiLoCo have demonstrated promising results, they often fail to fully exploit computational clusters under dynamic workloads. To address this limitation, we propose a three-stage method that combines Multi-Instance Training (MIT), Adaptive Batched DiLoCo, and switch mode mechanism. MIT allows individual nodes to run multiple lightweight training streams with different model instances in parallel and merge them to combine knowledge, increasing throughput and reducing idle time. Adaptive Batched DiLoCo dynamically adjusts local batch sizes to balance computation and communication, substantially lowering synchronization delays. Switch mode further stabilizes training by seamlessly introducing gradient accumulation once adaptive batch sizes grow beyond hardware-friendly limits. Together, these innovations improve both convergence speed and system efficiency. We also provide a theoretical estimate of the number of communications required for the full convergence of a model trained using our method.