8-bit Numerical Formats for Deep Neural Networks
This addresses the need for more efficient training and inference in large-scale machine learning models, though it is incremental as it builds on existing low-precision methods.
The paper tackled the problem of improving computational efficiency in deep neural networks by studying 8-bit floating-point formats for activations, weights, and gradients, and found that suitable low-precision formats enable faster training and reduced power consumption without accuracy degradation for image classification and language processing models.
Given the current trend of increasing size and complexity of machine learning architectures, it has become of critical importance to identify new approaches to improve the computational efficiency of model training. In this context, we address the advantages of floating-point over fixed-point representation, and present an in-depth study on the use of 8-bit floating-point number formats for activations, weights, and gradients for both training and inference. We explore the effect of different bit-widths for exponents and significands and different exponent biases. The experimental results demonstrate that a suitable choice of these low-precision formats enables faster training and reduced power consumption without any degradation in accuracy for a range of deep learning models for image classification and language processing.