Precision-Scalable Microscaling Datapaths with Optimized Reduction Tree for Efficient NPU Integration
This work addresses the need for efficient NPU platforms to support both training and inference in continual learning applications, representing an incremental improvement over existing MX designs.
The paper tackles the trade-off in existing Microscaling (MX) multiply-accumulate designs by proposing a hybrid precision-scalable reduction tree, enabling efficient mixed-precision accumulation with controlled accuracy relaxation. The integrated system achieves energy efficiencies of 657, 1438-1675, and 4065 GOPS/W for MXINT8, MXFP8/6, and MXFP4, respectively, with throughputs up to 512 GOPS.
Emerging continual learning applications necessitate next-generation neural processing unit (NPU) platforms to support both training and inference operations. The promising Microscaling (MX) standard enables narrow bit-widths for inference and large dynamic ranges for training. However, existing MX multiply-accumulate (MAC) designs face a critical trade-off: integer accumulation requires expensive conversions from narrow floating-point products, while FP32 accumulation suffers from quantization losses and costly normalization. To address these limitations, we propose a hybrid precision-scalable reduction tree for MX MACs that combines the benefits of both approaches, enabling efficient mixed-precision accumulation with controlled accuracy relaxation. Moreover, we integrate an 8x8 array of these MACs into the state-of-the-art (SotA) NPU integration platform, SNAX, to provide efficient control and data transfer to our optimized precision-scalable MX datapath. We evaluate our design both on MAC and system level and compare it to the SotA. Our integrated system achieves an energy efficiency of 657, 1438-1675, and 4065 GOPS/W, respectively, for MXINT8, MXFP8/6, and MXFP4, with a throughput of 64, 256, and 512 GOPS.