Mechanism-Driven Monitors for Preemptive Detection of LLM Training Instability
For large-scale LLM training, this work provides early warning of costly stability failures, potentially saving significant computational resources.
The paper introduces mechanism-driven monitors for preemptive detection of LLM training instability, achieving detection thousands of steps before loss divergence across low-precision attention, large learning-rate, and combined faults.
Frontier large language model training consumes massive accelerator fleets and long wall-clock computation, making stability failures costly when they occur. After a numerical or a hyperparameter fault has already destabilized the training dynamics, it may continue for thousands of steps while loss and gradient norms still appear normal. We study mechanism-driven detection of training instability by deriving internal monitors from the functional role of each critical module and from the earliest computational sites where failures are expected to produce measurable signatures. For low-precision flash attention, we monitor the spectral entropy of a QK bilinear decomposition, whose first-order term becomes abnormal before the loss fully collapses. For MoE routers, we derive indicators from their role in expert selection. Our fault-injection experiments on low-precision attention, large learning-rate, and combined faults show that these signals provide distinct signatures for different failures, triggering thousands of steps before loss divergence.