QuantNet: Learning to Quantize by Learning within Fully Differentiable Framework
This work addresses performance degradation in binary neural networks for efficient deep learning deployment, representing a novel method rather than an incremental improvement.
The paper tackles the problem of gradient mismatching and discretization errors in binary neural networks by proposing QuantNet, a meta-based quantizer that operates within a fully differentiable framework without using STE, achieving significant accuracy improvements on CIFAR-100 and ImageNet and bridging gaps with full-precision models.
Despite the achievements of recent binarization methods on reducing the performance degradation of Binary Neural Networks (BNNs), gradient mismatching caused by the Straight-Through-Estimator (STE) still dominates quantized networks. This paper proposes a meta-based quantizer named QuantNet, which utilizes a differentiable sub-network to directly binarize the full-precision weights without resorting to STE and any learnable gradient estimators. Our method not only solves the problem of gradient mismatching, but also reduces the impact of discretization errors, caused by the binarizing operation in the deployment, on performance. Generally, the proposed algorithm is implemented within a fully differentiable framework, and is easily extended to the general network quantization with any bits. The quantitative experiments on CIFAR-100 and ImageNet demonstrate that QuantNet achieves the signifficant improvements comparing with previous binarization methods, and even bridges gaps of accuracies between binarized models and full-precision models.