Sampling-Free Learning of Bayesian Quantized Neural Networks
This work addresses uncertainty calibration in quantized neural networks for scenarios like resource-constrained applications, representing an incremental improvement over existing methods.
The paper tackles the problem of learning Bayesian quantized neural networks (BQNs) to provide well-calibrated uncertainty estimates, proposing efficient sampling-free algorithms that achieve lower predictive errors and better-calibrated uncertainties than bootstrap ensembles of QNNs, with less than 20% of the negative log-likelihood on image classification datasets.
Bayesian learning of model parameters in neural networks is important in scenarios where estimates with well-calibrated uncertainty are important. In this paper, we propose Bayesian quantized networks (BQNs), quantized neural networks (QNNs) for which we learn a posterior distribution over their discrete parameters. We provide a set of efficient algorithms for learning and prediction in BQNs without the need to sample from their parameters or activations, which not only allows for differentiable learning in QNNs, but also reduces the variance in gradients. We evaluate BQNs on MNIST, Fashion-MNIST, KMNIST and CIFAR10 image classification datasets, compared against bootstrap ensemble of QNNs (E-QNN). We demonstrate BQNs achieve both lower predictive errors and better-calibrated uncertainties than E-QNN (with less than 20% of the negative log-likelihood).