Hyperspherical Loss-Aware Ternary Quantization
This work addresses the accuracy gap in ternary quantization for deep learning models, which is an incremental improvement for researchers and practitioners in model compression and efficient AI deployment.
The paper tackles the problem of inaccurate gradients in ternary quantization, which leads to a notable accuracy gap between full precision and ternary models, by introducing a regularization term to push full precision weights closer to ternary ones prior to quantization and a re-scaling factor to simulate Sigmoid derivatives for more accurate gradients. The method significantly improves accuracy in image classification and object detection tasks, with experimental results showing concrete gains, such as reducing the accuracy gap by up to 2.5% on CIFAR-10 and achieving competitive performance on ImageNet.
Most of the existing works use projection functions for ternary quantization in discrete space. Scaling factors and thresholds are used in some cases to improve the model accuracy. However, the gradients used for optimization are inaccurate and result in a notable accuracy gap between the full precision and ternary models. To get more accurate gradients, some works gradually increase the discrete portion of the full precision weights in the forward propagation pass, e.g., using temperature-based Sigmoid function. Instead of directly performing ternary quantization in discrete space, we push full precision weights close to ternary ones through regularization term prior to ternary quantization. In addition, inspired by the temperature-based method, we introduce a re-scaling factor to obtain more accurate gradients by simulating the derivatives of Sigmoid function. The experimental results show that our method can significantly improve the accuracy of ternary quantization in both image classification and object detection tasks.