Symmetry Regularization and Saturating Nonlinearity for Robust Quantization
This work addresses the challenge of reliable neural network deployment in low-precision hardware, offering incremental improvements to existing quantization techniques.
The paper tackled the problem of improving neural network robustness to quantization errors by proposing symmetry regularization and saturating nonlinearity methods, which enhanced performance across various bit-widths and algorithms, achieving competitive results on CIFAR and ImageNet datasets.
Robust quantization improves the tolerance of networks for various implementations, allowing reliable output in different bit-widths or fragmented low-precision arithmetic. In this work, we perform extensive analyses to identify the sources of quantization error and present three insights to robustify a network against quantization: reduction of error propagation, range clamping for error minimization, and inherited robustness against quantization. Based on these insights, we propose two novel methods called symmetry regularization (SymReg) and saturating nonlinearity (SatNL). Applying the proposed methods during training can enhance the robustness of arbitrary neural networks against quantization on existing post-training quantization (PTQ) and quantization-aware training (QAT) algorithms and enables us to obtain a single weight flexible enough to maintain the output quality under various conditions. We conduct extensive studies on CIFAR and ImageNet datasets and validate the effectiveness of the proposed methods.