SysOM-AI: Continuous Cross-Layer Performance Diagnosis for Production AI Training
This addresses performance inefficiencies for large-scale AI training operators, such as those at Alibaba, by enabling faster diagnosis of production issues across thousands of GPUs.
The paper tackled the problem of performance diagnosis in production-scale AI training, where subtle OS-level issues cause cascading delays, by introducing SysOM-AI, a continuous cross-layer observability system that reduced median diagnosis time from days to about 10 minutes and incurred less than 0.4% overhead.
Performance diagnosis in production-scale AI training is challenging because subtle OS-level issues can trigger cascading GPU delays and network slowdowns, degrading training efficiency across thousands of GPUs. Existing profiling tools are limited to single system layers, incur prohibitive overhead (10--30%), or lack continuous deployment capabilities, resulting in manual analyses spanning days. We argue that continuous, cross-layer observability enabled by OS-level instrumentation and layered differential diagnosis is necessary to address this gap. We introduce SysOM-AI, a production observability system that continuously integrates CPU stack profiling, GPU kernel tracing, and NCCL event instrumentation via adaptive hybrid stack unwinding and eBPF-based tracing, incurring less than 0.4% overhead. Deployed at Alibaba across over 80,000 GPUs for more than one year, SysOM-AI helped diagnose 94 confirmed production issues, reducing median diagnosis time from days to approximately 10 minutes.