What happens when nanochat meets DiLoCo?
This work addresses the problem of communication-constrained LLM training for distributed environments, but it is incremental as it builds on existing methods like DiLoCo and nanochat to explore trade-offs.
The paper studied the impact of using the DiLoCo algorithm for communication-efficient distributed LLM training on model performance, finding that while it achieved stable convergence and competitive loss in pretraining, it led to worse scores on MMLU, GSM8K, and HumanEval tasks after mid-training and SFT, with irreversible representation drift that impaired downstream alignment.
Although LLM training is typically centralized with high-bandwidth interconnects and large compute budgets, emerging methods target communication-constrained training in distributed environments. The model trade-offs introduced by this shift remain underexplored, and our goal is to study them. We use the open-source nanochat project, a compact 8K-line full-stack ChatGPT-like implementation containing tokenization, pretraining, fine-tuning, and serving, as a controlled baseline. We implement the DiLoCo algorithm as a lightweight wrapper over nanochat's training loop, performing multiple local steps per worker before synchronization with an outer optimizer, effectively reducing communication by orders of magnitude. This inner-outer training is compared against a standard data-parallel (DDP) setup. Because nanochat is small and inspectable, it enables controlled pipeline adaptations and allows direct comparison with the conventional centralized baseline. DiLoCo achieves stable convergence and competitive loss in pretraining but yields worse MMLU, GSM8K, and HumanEval scores after mid-training and SFT. We discover that using DiLoCo-pretrained weights and running mid- and post-training with DDP fails to recover performance, revealing irreversible representation drift from asynchronous updates that impairs downstream alignment. We provide this implementation as an official fork of nanochat on GitHub.