DynaTrain: Fast Online Parallelism Switching for Elastic LLM Training
For large-scale LLM training, DynaTrain solves the bottleneck of costly parallelism reconfiguration, enabling dynamic resource adaptation and elastic training.
DynaTrain enables sub-second online switching between arbitrary parallelism configurations during LLM training, achieving reconfiguration times of under 2s for a 70B dense model and 4.36s for a 235B MoE model, outperforming prior systems by up to three orders of magnitude.
Modern large language model (LLM) training is inherently dynamic: resource fluctuations, RLHF phase shifts, and cluster elasticity continually reshape the optimal parallelism layout, posing a significant challenge to existing training frameworks built around a static execution model. We present DynaTrain, a distributed training system for sub-second, online reconfiguration across arbitrary multi-dimensional parallelism. At its core, we propose a Virtual Parameter Space (VPS) abstraction that unifies all distributed training states under one logical coordinate space, turning any parallelism configuration into a deterministic mapping and collapsing complex transition into manageable geometric intersections. On top of VPS, a state routing-and-transition layer executes rank-local transfers under a memory-aware, deadlock-free schedule, and an Elastic Device Manager overlaps new-world construction with ongoing training to mask topology-change cost. On dense and MoE models up to 235B parameters, DynaTrain reconfigures a 70B dense model in under 2s and a 235B MoE model in 4.36s, outperforming state-of-the-art checkpoint-based and elastic systems by up to three orders of magnitude while preserving correctness.