Unleashing Low-Bit Inference on Ascend NPUs: A Comprehensive Evaluation of HiFloat Formats
This work addresses efficiency challenges in LLM inference on specific hardware, but it is incremental as it builds on existing low-bit formats and quantization methods.
The paper tackled the problem of low-bit inference for large language models on Ascend NPUs by evaluating HiFloat formats, finding that HiF4 prevents accuracy collapse in 4-bit regimes and is compatible with existing quantization frameworks.
As LLMs scale, low-bit floating-point formats like MXFP and NVFP4 offer new opportunities for precision and efficiency. In this work, we evaluate HiFloat (HiF8 and HiF4), a family of formats tailored for Ascend NPUs. Through rigorous comparison across weight-activation and KV-cache tasks, we provide three key insights: (1) INT8 suits narrow-range data, while floating-point formats excel with high-variance data; (2) in 4-bit regimes, HiF4's hierarchical scaling prevents the accuracy collapse seen in integer formats; and (3) HiFloat is fully compatible with state-of-the-art post-training quantization frameworks. Overall, HiFloat provides a solution for high-efficiency LLM inference on NPUs.