FP8 Formats for Deep Learning
This work addresses the need for more efficient deep learning computation for practitioners and researchers, though it is incremental as it builds on existing low-precision formats.
The paper tackles the problem of accelerating deep learning training and inference by proposing an 8-bit floating point (FP8) format with two encodings, E4M3 and E5M2, and demonstrates that it effectively matches the result quality of 16-bit training on various image and language tasks, including large models up to 175B parameters.
FP8 is a natural progression for accelerating deep learning training inference beyond the 16-bit formats common in modern processors. In this paper we propose an 8-bit floating point (FP8) binary interchange format consisting of two encodings - E4M3 (4-bit exponent and 3-bit mantissa) and E5M2 (5-bit exponent and 2-bit mantissa). While E5M2 follows IEEE 754 conventions for representatio of special values, E4M3's dynamic range is extended by not representing infinities and having only one mantissa bit-pattern for NaNs. We demonstrate the efficacy of the FP8 format on a variety of image and language tasks, effectively matching the result quality achieved by 16-bit training sessions. Our study covers the main modern neural network architectures - CNNs, RNNs, and Transformer-based models, leaving all the hyperparameters unchanged from the 16-bit baseline training sessions. Our training experiments include large, up to 175B parameter, language models. We also examine FP8 post-training-quantization of language models trained using 16-bit formats that resisted fixed point int8 quantization.