Exploring Silent Data Corruption as a Reliability Challenge in LLM Training
This addresses a reliability problem for LLM developers by providing a controlled analysis and mitigation for hardware-induced faults that can disrupt training, though it is incremental as it builds on existing fault injection techniques.
The study investigated how Silent Data Corruption (SDC) from hardware faults affects LLM pretraining, finding that it can cause loss spikes and parameter divergence, and proposed a detection method that, when applied to LLaMA models up to 1.3B parameters, effectively mitigated these issues by recomputing training steps.
As Large Language Models (LLMs) scale in size and complexity, the consequences of failures during training become increasingly severe. A major challenge arises from Silent Data Corruption (SDC): hardware-induced faults that bypass system-level detection mechanisms. SDC may behave like benign numerical noise, but can also cause harmful gradient corruption that leads to loss spikes, divergence, or stalled progress. This work provides a controlled study of how intermittent SDC affects LLM pretraining. Using targeted fault injection at the level of GPU matrix-multiply instructions, we characterize the sensitivity of different bit positions, kernel functions, and execution stages. Our analysis shows that locally originating faults can produce impactful corruption, including NaN propagation, short-lived spikes in loss, gradient norm, and attention logits, as well as persistent parameter divergence. Building on the observed corruption signatures, we propose a lightweight detection method that identifies potentially harmful parameter updates. Experiments on LLaMA models with 60M, 350M, and 1.3B parameters demonstrate that recomputing the most recent training step upon detection can effectively mitigate the impact of these events.