Toward INT4 Fixed-Point Training via Exploring Quantization Error for Gradients
This work addresses efficient training for deep learning models by enabling INT4 fixed-point training, which is incremental as it builds on prior gradient quantization methods.
The paper tackles the problem of low-bit fixed-point training by analyzing gradient quantization error, showing that reducing error for large-magnitude gradients significantly improves performance, and proposes an adaptive interval update algorithm. Experimental results demonstrate effectiveness across various network architectures, bit-widths, and tasks like image classification, object detection, and super-resolution.
Network quantization generally converts full-precision weights and/or activations into low-bit fixed-point values in order to accelerate an inference process. Recent approaches to network quantization further discretize the gradients into low-bit fixed-point values, enabling an efficient training. They typically set a quantization interval using a min-max range of the gradients or adjust the interval such that the quantization error for entire gradients is minimized. In this paper, we analyze the quantization error of gradients for the low-bit fixed-point training, and show that lowering the error for large-magnitude gradients boosts the quantization performance significantly. Based on this, we derive an upper bound of quantization error for the large gradients in terms of the quantization interval, and obtain an optimal condition for the interval minimizing the quantization error for large gradients. We also introduce an interval update algorithm that adjusts the quantization interval adaptively to maintain a small quantization error for large gradients. Experimental results demonstrate the effectiveness of our quantization method for various combinations of network architectures and bit-widths on various tasks, including image classification, object detection, and super-resolution.