FP8 Quantization: The Power of the Exponent
This work addresses the challenge of efficient neural network deployment for practitioners by showing that FP8 quantization can improve accuracy in post-training scenarios, though it is incremental as it builds on existing quantization methods.
The paper tackled the problem of quantizing neural networks for efficient inference by investigating the benefits of using 8-bit floating point (FP8) formats over integer formats, finding that FP8 yields better accuracy than INT8 for post-training quantization across various networks, with the optimal number of exponent bits determined by outlier severity.
When quantizing neural networks for efficient inference, low-bit integers are the go-to format for efficiency. However, low-bit floating point numbers have an extra degree of freedom, assigning some bits to work on an exponential scale instead. This paper in-depth investigates this benefit of the floating point format for neural network inference. We detail the choices that can be made for the FP8 format, including the important choice of the number of bits for the mantissa and exponent, and show analytically in which settings these choices give better performance. Then we show how these findings translate to real networks, provide an efficient implementation for FP8 simulation, and a new algorithm that enables the learning of both the scale parameters and the number of exponent bits in the FP8 format. Our chief conclusion is that when doing post-training quantization for a wide range of networks, the FP8 format is better than INT8 in terms of accuracy, and the choice of the number of exponent bits is driven by the severity of outliers in the network. We also conduct experiments with quantization-aware training where the difference in formats disappears as the network is trained to reduce the effect of outliers.