Neural Networks with Quantization Constraints
This work addresses the need for efficient deep learning in resource-constrained settings, offering a novel method for quantization-aware training that is incremental but provides specific gains.
The paper tackles the problem of enabling low-precision deep learning models without significant performance loss by formulating quantization-aware training as a constrained optimization problem, showing it avoids gradient approximations and achieves competitive performance in image classification, with layer selective quantization leading to considerable improvements.
Enabling low precision implementations of deep learning models, without considerable performance degradation, is necessary in resource and latency constrained settings. Moreover, exploiting the differences in sensitivity to quantization across layers can allow mixed precision implementations to achieve a considerably better computation performance trade-off. However, backpropagating through the quantization operation requires introducing gradient approximations, and choosing which layers to quantize is challenging for modern architectures due to the large search space. In this work, we present a constrained learning approach to quantization aware training. We formulate low precision supervised learning as a constrained optimization problem, and show that despite its non-convexity, the resulting problem is strongly dual and does away with gradient estimations. Furthermore, we show that dual variables indicate the sensitivity of the objective with respect to constraint perturbations. We demonstrate that the proposed approach exhibits competitive performance in image classification tasks, and leverage the sensitivity result to apply layer selective quantization based on the value of dual variables, leading to considerable performance improvements.