Understanding Silent Data Corruption in LLM Training
This addresses the problem of hardware-induced errors in LLM training for researchers and practitioners, but it is incremental as it builds on existing understanding of SDCs.
The paper investigates the impact of silent data corruption (SDC) on large language model (LLM) training by comparing healthy and unhealthy production nodes, revealing that SDCs can cause models to converge to different optima and lead to spikes in training loss.
As the scale of training large language models (LLMs) increases, one emergent failure is silent data corruption (SDC), where hardware produces incorrect computations without explicit failure signals. In this work, we are the first to investigate the impact of real-world SDCs on LLM training by comparing model training between healthy production nodes and unhealthy nodes exhibiting SDCs. With the help from a cloud computing platform, we access the unhealthy nodes that were swept out from production by automated fleet management. Using deterministic execution via XLA compiler and our proposed synchronization mechanisms, we isolate and analyze the impact of SDC errors on these nodes at three levels: at each submodule computation, at a single optimizer step, and at a training period. Our results reveal that the impact of SDCs on computation varies on different unhealthy nodes. Although in most cases the perturbations from SDCs on submodule computation and gradients are relatively small, SDCs can lead models to converge to different optima with different weights and even cause spikes in the training loss. Our analysis sheds light on further understanding and mitigating the impact of SDCs.