Parallelizing Large-Scale Tensor Network Contraction on Multiple GPUs
This work addresses the scalability bottleneck of exact tensor network contraction for quantum circuit simulation and related fields, demonstrating that communication-aware distribution far surpasses slicing-based scaling limits.
The paper presents a multi-GPU framework for exact tensor network contraction that distributes intermediate tensors across devices with explicit communication, achieving 7-173x extra speedup beyond slicing on a single DGX H100 node and 42-67,869x on 1024 H100 GPUs over InfiniBand.
Exact tensor network contraction underpins quantum circuit simulation, quantum error correction, combinatorial optimization, and many-body dynamics. The dominant parallelization strategy, slicing, scales exponentially and incurs redundant computation. We present a multi-GPU framework that instead distributes intermediate tensors across devices with explicit communication, converting a fixed contraction path into a communication-efficient schedule via GEMM-oriented mode reordering and communication-aware mode distribution planning. Within a single DGX H100 node (8 GPUs, NVLink), distribution delivers $7$--$173\times$ extra speedup beyond embarrassingly parallel slicing, capturing nearly all of the available compute reduction (87--101%) because NVLink's high bandwidth keeps communication small relative to compute. Scaling the same four workloads to 1024 H100 GPUs over InfiniBand, the extra speedup beyond slicing ranges from $42\times$ to $67{,}869\times$, demonstrating that communication-aware distributed contraction far surpasses slicing-based scaling limits for frontier tensor networks.