EmuGEMM: Fused Tensor Core Kernels for Precision Emulation in Matrix Multiplication
For scientific computing workloads requiring high-precision GEMM on modern GPUs, EmuGEMM provides a fused kernel approach that significantly improves throughput over existing implementations.
EmuGEMM fuses integer Tensor Core kernels to eliminate redundant memory round-trips in Ozaki precision emulation schemes, achieving up to 1,639 Top/s on Hopper (83% of INT8 peak) and surpassing cuBLAS TF32 throughput by 1.4x on Hopper and 1.7x on Blackwell for large matrices, with comparable accuracy.
Modern GPUs devote an increasing silicon budget to low-precision matrix-multiplication units, widening the precision-throughput gap for scientific computing workloads. Ozaki Schemes I and II offer an alternative by reconstructing high-precision general matrix multiplication (GEMM) from low-precision operations, yet existing implementations leave substantial performance untapped. In particular, intermediate results are repeatedly materialized in global memory, making data movement the dominant bottleneck. We present EmuGEMM, fused integer Tensor Core kernels for NVIDIA Hopper and Blackwell GPUs that eliminate redundant memory round-trips in both Ozaki schemes. Using Scheme I, EmuGEMM sustains up to 1,639 Top/s on Hopper (83% of INT8 peak) and 3,654 Top/s on Blackwell (81%). For large matrices, EmuGEMM surpasses cuBLAS TF32 throughput by up to 1.4x on Hopper and 1.7x on Blackwell, at comparable accuracy. Using Scheme II, EmuGEMM extends to complex arithmetic and outperforms cuBLAS ZGEMM by up to 2.3x on Hopper and 5.5x on Blackwell.