Sustaining Exascale Performance: Lessons from HPL and HPL-MxP on Aurora
It addresses performance sustainability for exascale computing in production environments, with incremental lessons applicable to heterogeneous systems at extreme scale.
The paper tackled the challenge of sustaining exascale performance on the Aurora system, achieving 1.01 EF/s on 9,234 nodes for FP64 HPL and 11.64 EF/s for HPL-MxP, an 11.5x speedup using mixed-precision arithmetic and Intel AMX acceleration.
Sustaining exascale performance in production requires engineering choices and operational practices that emerge only under real deployment constraints and demand coordination across system layers. This paper reports experience from three successive campaigns running HPL and HPL-MxP on Aurora, an Intel-based exascale system featuring the first large-scale deployment of Intel discrete GPUs, CPU-attached network interfaces, and the largest production Slingshot-11 interconnect. Aurora progressed from 0.585EF/s on 5,439 nodes to 1.01EF/s on 9,234 nodes in FP64 HPL, while HPL-MxP reached 11.64EF/s, an 11.5x speedup over FP64 enabled by mixed-precision arithmetic and Intel AMX acceleration. We identify and classify by role at production scale the system-level choices that sustained these results, including deterministic locality-aware resource mapping, explicit CPU-GPU pipelining, mixed-precision orchestration, and a hybrid P2P/collective resilience strategy introduced after synchronization stalls at scale. While some observations are Aurora-specific, the broader lessons are likely to apply to tightly coupled heterogeneous systems at extreme scale.