MMA-Sim: Bit-Accurate Reference Model of Tensor Cores and Matrix Cores
This addresses the need for accurate simulation of undocumented hardware behaviors to ensure stable and reproducible deep neural network training and inference, though it is incremental as it builds on existing hardware analysis methods.
The paper tackles the problem of numerical imprecision and inconsistency in matrix multiplication accelerators (MMAs) like NVIDIA Tensor Cores and AMD Matrix Cores, which can compromise DNN stability and reproducibility, by presenting MMA-Sim, a bit-accurate reference model that simulates these accelerators with bitwise equivalence validated on ten GPU architectures.
The rapidly growing computation demands of deep neural networks (DNNs) have driven hardware vendors to integrate matrix multiplication accelerators (MMAs), such as NVIDIA Tensor Cores and AMD Matrix Cores, into modern GPUs. However, due to distinct and undocumented arithmetic specifications for floating-point matrix multiplication, some MMAs can lead to numerical imprecision and inconsistency that can compromise the stability and reproducibility of DNN training and inference. This paper presents MMA-Sim, the first bit-accurate reference model that reveals the detailed arithmetic behaviors of the MMAs from ten GPU architectures (eight from NVIDIA and two from AMD). By dissecting the MMAs using a combination of targeted and randomized tests, our methodology derives nine arithmetic algorithms to simulate the floating-point matrix multiplication of the MMAs. Large-scale validation confirms bitwise equivalence between MMA-Sim and the real hardware. Using MMA-Sim, we investigate arithmetic behaviors that affect DNN training stability, and identify undocumented behaviors that could lead to significant errors.