Multi-GPU Hybrid Particle-in-Cell Monte Carlo Simulations for Exascale Computing Systems
This work addresses performance bottlenecks for plasma physics simulations on modern HPC systems, representing an incremental advancement in optimizing computational methods for exascale computing.
The paper tackled the challenges of Particle-in-Cell Monte Carlo simulations on heterogeneous HPC systems by developing a portable, multi-GPU hybrid implementation, resulting in significant improvements in run time, scalability, and resource utilization on exascale systems like Frontier with up to 16,000 GPUs.
Particle-in-Cell (PIC) Monte Carlo (MC) simulations are central to plasma physics but face increasing challenges on heterogeneous HPC systems due to excessive data movement, synchronization overheads, and inefficient utilization of multiple accelerators. In this work, we present a portable, multi-GPU hybrid MPI+OpenMP implementation of BIT1 that enables scalable execution on both Nvidia and AMD accelerators through OpenMP target tasks with explicit dependencies to overlap computation and communication across devices. Portability is achieved through persistent device-resident memory, an optimized contiguous one-dimensional data layout, and a transition from unified to pinned host memory to improve large data-transfer efficiency, together with GPU Direct Memory Access (DMA) and runtime interoperability for direct device-pointer access. Standardized and scalable I/O is provided using openPMD and ADIOS2, supporting high-performance file I/O, in-memory data streaming, and in-situ analysis and visualization. Performance results on pre-exascale and exascale systems, including Frontier (OLCF-5) for up to 16,000 GPUs, demonstrate significant improvements in run time, scalability, and resource utilization for large-scale PIC MC simulations.