LLM-PRISM: Characterizing Silent Data Corruption from Permanent GPU Faults in LLM Training
This study provides the first hardware-grounded characterization of SDC resilience in LLM pre-training, addressing a critical reliability issue for large-scale training.
LLM-PRISM characterizes LLM pre-training resilience to permanent GPU faults causing Silent Data Corruption, finding that while low-frequency faults are resisted, critical datapaths and specific precision formats can cause catastrophic divergence even at moderate fault rates.
Large-scale LLM training is increasingly susceptible to hardware defects stemming from manufacturing escapes and silicon aging. These defects manifest as Silent Data Corruption (SDC) that perturb gradients and parameters throughout the training process. We present LLM-PRISM, a methodology to characterize LLM pre-training resilience to hardware faults. LLM-PRISM couples RTL-level GPU fault simulation with a stochastic injection engine embedded in Megatron-LM. Through 7,664 training runs across FP16, BF16, and FP8 regimes, we analyze how fault type, rate, and numeric format govern resilience. We find that while LLMs resist low-frequency faults, impact is highly non-uniform; critical datapaths and specific precision formats can induce catastrophic divergence even at moderate fault rates. This study provides the first hardware-grounded, pre-training characterization of SDC resilience.