PROFIT: A Novel Training Method for sub-4-bit MobileNet Models
This work addresses the need for energy-efficient mobile models by enabling lower-precision quantization, though it is incremental as it builds on existing quantization techniques.
The paper tackles the challenge of sub-4-bit quantization for mobile networks by identifying activation instability induced by weight quantization as a key obstacle, and proposes the PROFIT training method to address it, achieving within 1.48% top-1 accuracy for 4-bit MobileNet models on ImageNet and a 12.86% improvement for 3-bit MobileNet-v3 over state-of-the-art methods.
4-bit and lower precision mobile models are required due to the ever-increasing demand for better energy efficiency in mobile devices. In this work, we report that the activation instability induced by weight quantization (AIWQ) is the key obstacle to sub-4-bit quantization of mobile networks. To alleviate the AIWQ problem, we propose a novel training method called PROgressive-Freezing Iterative Training (PROFIT), which attempts to freeze layers whose weights are affected by the instability problem stronger than the other layers. We also propose a differentiable and unified quantization method (DuQ) and a negative padding idea to support asymmetric activation functions such as h-swish. We evaluate the proposed methods by quantizing MobileNet-v1, v2, and v3 on ImageNet and report that 4-bit quantization offers comparable (within 1.48 % top-1 accuracy) accuracy to full precision baseline. In the ablation study of the 3-bit quantization of MobileNet-v3, our proposed method outperforms the state-of-the-art method by a large margin, 12.86 % of top-1 accuracy.