LEO: Tracing GPU Stall Root Causes via Cross-Vendor Backward Slicing
This addresses a diagnostic gap for developers on GPU-accelerated supercomputers, enabling more efficient code optimization, though it is incremental as it builds on existing profiling and slicing techniques.
The paper tackles the problem of diagnosing GPU stall root causes across different GPU vendors by presenting LEO, a root-cause analyzer that performs backward slicing from stalled instructions, resulting in geometric-mean speedups of 1.73x to 1.82x across 21 workloads on three GPU platforms.
More than half of the Top 500 supercomputers employ GPUs as accelerators. On GPU-accelerated platforms, developers face a key diagnostic gap: profilers show source lines where stalls occur, but not why they occur. Furthermore, the same kernel may have different stalls and underlying causes on different GPUs. This paper presents LEO, a root-cause analyzer for NVIDIA, AMD, and Intel GPUs that performs backward slicing from stalled instructions, considering dependencies arising from registers as well as vendor-specific synchronization mechanisms. LEO attributes GPU stalls to source instructions with the goal of explaining root causes of these inefficiencies. Across 21 workloads on three GPU platforms, LEO-guided optimizations deliver geometric-mean speedups of 1.73$\times$--1.82$\times$. Our case studies show that (1) the same kernel may require different optimizations for different GPU architectures, and (2) LEO's structured diagnostics improve code optimization with large language models relative to code-only and raw-stall-count baselines.