On Calibration of Modern Quantized Efficient Neural Networks
This work addresses calibration issues for quantized networks, which is important for reliable EdgeML, but it is incremental as it builds on known quantization and calibration methods.
The paper investigated calibration properties of quantized neural networks across architectures and datasets, finding that calibration quality degrades with lower precision, especially at 4-bit activations, and that temperature scaling can improve calibration error with limitations.
We explore calibration properties at various precisions for three architectures: ShuffleNetv2, GhostNet-VGG, and MobileOne; and two datasets: CIFAR-100 and PathMNIST. The quality of calibration is observed to track the quantization quality; it is well-documented that performance worsens with lower precision, and we observe a similar correlation with poorer calibration. This becomes especially egregious at 4-bit activation regime. GhostNet-VGG is shown to be the most robust to overall performance drop at lower precision. We find that temperature scaling can improve calibration error for quantized networks, with some caveats. We hope that these preliminary insights can lead to more opportunities for explainable and reliable EdgeML.